this_blog(cool, exciting) :- posts(kleetus)

How to fix -- dlopen(/Users/user/.rvm/gems/ruby-1.9.2-p290@gemset/gems/do_mysql-0.10.7/lib/do_mysql/do_mysql.bundle, 9): Library not loaded

2012-01-16T16:50:00.000-08:00

Recently when working with a new gemset in RVM, I noticed that the gem install of mysql-0.10.7, the C extensions be exact, did not link correctly against the mysqllibclient dynamic library. I don't really know why this at this point. I needed to get it fixed. Stackoverflow to the rescue, but I wanted to post the script I made to make this easier.

The error when running my Sinatra app:

dlopen(/Users/users/.rvm/gems/ruby-1.9.2-p290@gemset/gems/do_mysql-0.10.7/lib/do_mysql/do_mysql.bundle, 9): Library not loaded: libmysqlclient.18.dylib (LoadError)

The ruby script for the fix:


#!/bin/env ruby
mysql_bundle = `find ~ -name do_mysql.bundle`
mysql_lib = 'libmysqlclient.18.dylib'
mysql_lib_location = `find /usr/local -name #{mysql_lib}`
m_array = mysql_lib_location.split
mysql_lib_path = m_array.first if m_array
mysql_bundle.split.each do |bundle|
`/usr/bin/install_name_tool -change #{mysql_lib} #{mysql_lib_path} #{bundle}`
end

Hey Paypal!

2011-08-18T16:24:00.000-07:00

PayPal wanted a survey from me..here is my reply. Wondered if this is too harsh?
"I periodically change all my passwords and my answers to the secret questions. This is a prudent action on my part, but apparently this behavior triggers an event related to a possible security problem in PayPal's system. I understand why this happens, but the security policies should be refactored. PayPal ought to use two factor authentication. For those that do not have access to the second factor, then drop back to the antiquated system. Also, if I remember correctly, PayPal does not allow passwords to be over a certain length. This is a super huge red flag that you are possibly storing my password plain text OR the frontend is just bush league and not sophisticated enough to prevent an overflow into the DB. In any case, this password policy is not what I expect from a banking institution. I don't have anything personal against your company, but it seems PayPal's priorities are mixed up."

SheevaPlug Dev Kit and Debian

2011-06-23T04:39:00.000-07:00

Got my Sheeva Plug Dev Kits in yesterday:

Initial thoughts are that they are nifty. Not earth-shattering. Not knee-buckling, just nifty.

In case you didn't know. The Sheeva Plug device is the hardware reference platform for a lot of pluggable computers these days such as the Tonido Plug and others. They have middle-of-the-road hardware specs based on the ARM architecture. The best part of this device is that the Dev kit retails out for $98. I ordered mine from GlobalScale Technologies.

GlobalScale took over two weeks just to ship out the two plugs that I ordered. Not cool. But, they did get them to me. The devices come ready to plug right into a standard wall 120V outlet. You can also remove the prongs from the device and plug in a power cable (sort of like the macbook's power transformer), so that you can mount the plug separate from the wall.

Input/Output connections to the device are:

Power (takes 120V directly, the power supply is in the device, interesting design decision)
MMC/SD slot
Gigabit ethernet
USB 2.0 port (A-type connection)
mini USB

This is the serial port to the device. You attach the supplied mini USB cable and there is a serial to USB device in the Sheeva Plug that gives you a serial connection. You can connection right up to the Sheeva Plug by doing something like:

% screen /dev/ttyUSB1 115200

Hardware notes:

Internal switching power supply. This is non-characteristic of hardware vendors these days. Let power supply doods handle that stuff, too much to go wrong there, but it does fit with the "plug" theme that they are going for. The MMC/SD is pretty standard, but I never really liked that form factor. The card sticks out about a full centimeter. This will snap off like a saltine at the hint of contact with something harder than talcum powder. Be careful if you use that port. Plenty of room around the usb port and ethernet port, so you can use fatty usb storage devices. The white protective case seems cheap and flimsy. I would not trust it to withstanding any sort of fall. But, this is reference hardware, not destined for the Apple Store.

Firmware notes:

When you connect to the serial console on the device, you will see the

Marvell>>

prompt. Consider this the bootloader for the device. The device has 512MB of internal flash memory, but this can't be used for the Debian install. I am not sure why this is, but I will investigate this as soon as I can and get back to you. As such, you will need to find another persistence data store. Common setups are installing Debian to the MMC or USB stick, but you could run everything from the network too. I bought these devices to run as small form factor home automation servers. So, I think the MMC/SD slot is perfect for the storage device. You can then use this link to install Debian onto the plug.

Remember: if you are writing software for the device, it architecture is "armel". The C/C++ libraries have different capabilities (they try to be the same), but you can easily cross compile for the Sheeva Plug from your amd64 machine (mostly everyone has one of those machines now, right?). Cross compiling software and directions can be found here

Setting all that up can be a bit of a pain. One of those things that drive you mad, but after going through all that, you will understand Unix guys have those giant beards.

Overall: I give the plug a 8.5/10.

Pros:

Open platform
Good hardware design (placement)
Inexpensive

Cons:

Cheap plastic case
Internal switching power supply makes me a bit nervous. If it goes bad, the device is toast.
MMC/SD slot needs to be one those spring-loaded deals (like cell phones use)...and it is OK to use microSD (only) at this point since those cards are all over the place.

I will be revising my review only the next couple of months when these plugs are attached to wall and doing their thing quietly. If I forget that they are there, then this will yield the best review possible.

In the coming weeks, I will be posting pictures of the device and how I am using it in the wild. One note, I tried to use a microSD in a full SD card insert to no available. The device was recognized, but the format of the device failed. I tried a full size SD 4GB card and this worked perfectly. SD cards are cheap and I don't need a huge amount of space on them. 4GB is plenty for a home automation server. As the days go on, I am liking this device more and more.

New favorite Microchip PIC is the 18F14K22

2010-12-29T05:01:00.000-08:00

I discovered this little microchip recently. The 18F14K22 PIC is really neat. At about $2 from mouser or digikey and supported by the USBProg for ICSP and also supported by mikroC compilers, it is my new favorite low-end chip. I use it anywhere USB coms are not required. It also runs at 64 MHz with an external crystal, but I mostly use the 2 internal clock sources. The primary internal runs at up to 16 MHz and has fantastic stability. The secondary clock runs at 31 KHz, which is great for saving the batteries. This chip comes in SMD/SMT and PDIP flavors as you might expect. I like the SOIC-20 style because this is really easy to solder down quickly on a prototype board (like a schmartboard). You can also run them at 1.8v to 5.5v, which is nice is you are using usb power or 2 double A batteries. I use this chip with the Linx TRM433 transceiver, which runs at a strict 3v. So I regulate power once and smooth it out and forget about it! If you are getting into the embedded world, I would highly recommend this chip! It beats the pants off of any arduino out there at a fraction of the cost!

Git: What SHA does this tag refer to?

2010-10-13T04:30:00.000-07:00

I often have to tag a branch in my Git repo for QA. Our QA people prefer to have a tag to deploy on to their QA environments, so that they do not have a "moving target" to ensure features and bugs are accounted for. Sometimes, however, I forget exactly where in the lineage the tag was created. In other words, on which commit the tag was drawn.

There is a way in Git to see this information. Here is a use case for the situation.

You complete feature A and commit it. You then tag feature A with an annotated tag and immediately start working on feature B, then C, etc. QA asks about the tag. You no longer remember what that tag was for, so you fire up a console and type in the following to find out:

git show-ref --dereference

You can find the tag in the list and perform a simple git log matching the sha from the show-ref to the sha in the git log. Simple. Even easier would be to run a filter using --exclude-existing on the show-ref using a regular expression and then piping the result to git show or git log.

Debian Unstable (SID) Squeeze: installing Grub2

2010-10-10T17:12:00.000-07:00

This is frustrating. Installing the Debian system in development (SID, AKA squeeze) is really broken currently. Installing GRUB or GRUB2 just does not work at all and can ruin your day. You get all the way to the end of the install when the error pops up. During the install of either GRUB or GRUB2.

Here is what you can do to fix this:

1. Choose to install GRUB2 and let it fail; be sure to allow GRUB to write to the MBR.
2. Go back to the main menu of installer items and choose "continue without installing a bootloader". Trust me on this one, it will all work out.
3. reboot your machine, the grub (this will be the GRUB2) bootloader prompt will be presented. This is your ticket back into your newly installed system
4. At the "grub>" prompt, type the following: (you may need to adjust the boot drive)

linux hd(0,1)/boot/vmlinuz-2.6.32-5-amd64 root=/dev/sda1

then:

initrd /boot/initrd-2.6.32-5-amd64

then:

boot

of course substitute the kernel and initrd you are using for the real ones. Also check the root drive for the correct one. Sometimes you can just guess, the machine will just not boot if it is wrong and you can go back and do it again until you get it right.

Then once you have everything booted. You can install GRUB2 properly by doing the following:

apt-get install grub2

then:

grub-install /dev/sda

then:

grub-mkconfig > /boot/grub/grub.cfg

then:

reboot

All should be well after the reboot.

Changing xterm-new/xterm text output colors

2010-09-29T10:45:00.000-07:00

I recently found that in my iterm (and Mac terminal) applications the text coloring was not optimal. For instance, I use a dark background, but when issuing a command such as 'ls' would list directories in dark blue. This is impossible to read. It seemed to me that this would affect most who prefer a dark background in their terminals. I think this only applies to Mac terminal since the environment variable used is a bit different than what Linux uses.

to find out which terminal you are using (aside from the application running the terminal such as terminal app or iterm) issue this command:

echo $TERM

as always, you can read all about the information here in the ls man page

man ls

here is what you can do to change one or all of your lscolors, environment variable is LSCOLORS

Add the following to .bashrc or .bash_profile or whatever file runs when you start an interactive shell: (mine is called .bash_profile and lives in my home directory)

LSCOLORS='fxfxcxdxbxegedabagacad'
export LSCOLORS

straight from 'man ls'

LSCOLORS
The value of this variable describes what color to use for which attribute when colors are enabled with CLICOLOR. This string is a concatenation of pairs of the format fb, where f is the foreground color and b is the background color.
The color designators are as follows:

a black
b red
c green
d brown
e blue
f magenta
g cyan
h light grey
A bold black, usually shows up as dark grey
B bold red
C bold green
D bold brown, usually shows up as yellow
E bold blue
F bold magenta
G bold cyan
H bold light grey; looks like bright white
x default foreground or background

Note that the above are standard ANSI colors. The actual display may differ depending on the color capabilities of the terminal in use.

The order of the attributes are as follows:

1. directory
2. symbolic link
3. socket
4. pipe
5. executable
6. block special
7. character special
8. executable with setuid bit set
9. executable with setgid bit set
10. directory writable to others, with sticky bit
11. directory writable to others, without sticky bit

The default is "exfxcxdxbxegedabagacad", i.e. blue foreground and default background for regular directories, black foreground and red background for setuid executables, etc.

So add the following to your .bash_profile or equivalent:

LSCOLORS='fxfxcxdxbxegedabagacad'
export LSCOLORS

As you can see I just changed the first character from 'e' to 'f'. This has the effect of making directory listings be listed in a magenta color instead of a dark blue.

Save .bash_profile and issue:

. !$

that is the same as:

source ~/.bash_profile

if issued right after the command to edit .bash_profile. In other words, you are asking bash to source (which is what the dot is) and then reprint the last argument to the last command.

Microchip Pic 18F2455/2550/4455/4550 Oscillator Settings Explained

2010-09-28T05:13:00.000-07:00

I have a new fav Microprocessor. The Microchip Pic 18F2455. This is a feature-full yet powerful little device has almost limitless possibilities. Alas, you must understand that I am coming from the Pic 16F77, which lacks USB support, but is far more simple to get "blinky lights".

What the 18F2455 achieves in features, it gives up in ease of use (for newbies like myself). The external oscillator settings where an epic nightmare to get right! I believe the reason for this is twofold:

1. Nanowatt (tm) specifies the use of 2 clocks/oscillators for the 18F2455. The first clock for the main clock source to run the cpu itself; the ALU, the USB port(s) and all that. The second clock runs timers. This setup allows the main clock (and CPU) to power down leaving the second clock running to handle checking for new data on the USB port and running timers.

2. The USB port needs a strict 48 MHz clock to run, but Microchip still wanted to allow users of these chips to use oscillator configurations that they were used to. For example, for me, I have always clocked my chips at 20 MHz and I have a surplus of those crystals, so using a 20 MHz clock source would be really nice. I can still do this, but I need to use internal chip configurations to make use of the chip's internal pre/post oscillator scalers.

Here is how it all breaks down for me:

using 18F2455 and a 20 MHz external crystal. Please open the data sheet for the 2455 and refer to figure 2-1.

diagraming it out:

20 MHZ --> pins 9,10 (on pic)

you can see from Figure 2-1 that whatever you hook up to pins 9,10, you are feeding a PLL prescaler internally and you MUST feed the subsequent PLL prescaler a 4 MHz clock source as its input. This means that for me, I have to prescale my 20 MHz oscillator to 4 MHz. This means that we need to divide 20 MHz by 5. You can see from figure 2-1 that this bit setting is 100 on the PLL DIV MUX.

Now you have 96 MHz at the exit of the second PLL that is feeding another divider and also feeding a PLL postscaler. So then you can set the CPU DIV for the appropriate div for the main CPU clock source. This is 1 in my case because I want the main CPU to be clocked at 20 MHz.

If you are thinking that this is pretty confusing, you are right! There are a lot of choices (12 different oscillator modes) in oscillator setup and configuration. This leads to confusion and dismay. Mayhem might ensue. But, I think if you take the time to read Fig. 2-1 and learn the register bits that control which mode the clocks can be in, then I think this is an extremely powerful tool.

Montana Property

2010-09-26T16:22:00.001-07:00

I wish I had the money and freedom to buy this property myself.

Check this beautiful Western USA property out!

52 acres for sale in Noxon Montana

SSH: setting up key authorization only (authorized_keys)

2010-09-20T05:12:00.000-07:00

This is a great site for dealing ssh key authorization.

http://www.eng.cam.ac.uk/help/jpmg/ssh/authorized_keys_howto.html

The problem with sites that help you setup ssh key authorization is that they typically leave out the specific chmod commands (permissions) that you need to make the darn thing work!

What I mean to say is that people need to see an example of how this works and not just out of context commands and scripts.

Here is an example of a Mac OS X ssh client trying to setup ssh key authorization with a CentOS ssh server.

on the Mac, issue the following commands (you may or may not want a pass phrase on your private key), also substitute your username and hostname in where the double quotes are.

1. ssh-keygen -t dsa

2. scp .ssh/id_dsa.pub "user"@"CentOS host":~

3. ssh "user"@"CentOS host"

on the CentOS server

1. mkdir .ssh (this directory may or may not already be there; it should be safe to issue either way)

2. cat id_dsa.pub > .ssh/authorized_keys

3. chmod 755 .ssh && chmod 755 .ssh/authorized_keys

on Mac again: (hit ctrl-D to log off the server)

1. ssh -vvv "user"@"CentOS host"

The ssh client should log you in without a password if there is not password protected private key. If you protected your private key with a password, then it should ask for the password for the private key. Ssh should NOT ask you to provide a login for the shell on the remote side. The triple 'V' will display a verbose list of the conversation going on betwix client and server.

Modulo operators with respect to Ruby and Javascript

2010-05-19T16:46:00.000-07:00

Recently, I was tasked with creating a calendar which transitioned from month to month such as December to January to February, etc.. Also, I needed to go backward on the transition.

A good way to do this is to use a modulo operator. This means that you can figure out which month is the next month number very easily.

So if the month is December, which is month number 12, to find out which month is next:

month = (month % 12) + 1 or (12 % 12 )+ 1, this will work for any month number from 1 to 12.

BUT to go backwards on the month numbers, here is the operation:

month = ((12+(month-2)) % 12) + 1 ... the funny thing is that this operation does not work in Ruby, but it does in Javascript.

In Ruby, here is the equivalent backward operation:

month = ((month-2) % 12) + 1

What is the difference? Why would there be a difference? I have no idea. It seems like Javascript would generate the right response to -1 % 12, which is -1 because 12 divided by -1 is 0 with -1 left over, but Ruby, Python, Perl, and the Google calculator figure that it should be 11 left over. This may be because a negative remainder would make no sense. So if the modulo is negative, then add it to the second operand to get the real modulo.

I must speak out on the this one

2010-04-22T14:26:00.000-07:00

Recently, while at work, I was discussing why Google maps do not include a scale. In fairness to Google, the maps may have this functionality, but I am ignorant as to how to enable this feature.

My hypothesis as to why there is no scale was a simple one. To accurately determine scale for a map, one would need to consider the topography of land the map covers. Maybe Google just does not want to go to the trouble.

Seems simple enough to me, but to my surprise, the other two parties in the conversation said that topography has nothing to do with scale on this map. Huh?!? I stipulated the point to avoid a nasty argument where more than one person might need to get schooled, Sesame Street style.

Well, I can't take it anymore; time for some schooling. The assertion I am attempting to buttress is that topography (whether there are mountains and valleys on the surface of the earth) matters when applying a scale marker on a map. Google could probably estimate this very accurately since topography overlays are available to them, but who wants to do calculus for such a small thing? Anyway, here is an image which details my explanation.

Installing Rails 3 on Debian Unstable (Sid)

2010-04-05T19:01:00.000-07:00

Recently, I was installing Rails3 on my Debian unstable machine and I noticed an error that was really strange.

Here was the error:

ERROR NameError: uninitialized constant ActiveResource::Base

The problem ended up being that I was missing this deb file:

libopenssl-ruby1.8

I have seen posts from people proclaiming that uninstalling old versions of activeresource would do the trick, but I reinstalled activeresource-2.3.5 gem after Rails 3 starting working again without a failure. I hate cryptic errors like that.

Mac OS X Leopard installing the ruby gem eventmachine

2010-03-21T18:38:00.000-07:00

I was having some trouble installing eventmachine (a ruby gem that needs an extension library written in C++) recently in Mac OS X Leopard. I am currently running ruby 1.8.8dev. I think, perhaps, when I compiled ruby from source, the config left out the CXX flags setting. This makes mkmf not know what C++ compiler it needs to set in the Makefiles it creates. Since there are not too many gems that have extensions written in C++ (that I know of), I never really ran into this issue. Here is how to fix this.

Edit this file inside the directory of the gem, in my case;

$ vim /usr/local/lib/ruby/gem/1.8/gem/eventmachine-0.12.11/ext/extconf.rb

Right above the last line (the last line being a call to create_makefile), insert this line:

CONFIG['CXX'] = g++

then do the same thing for this extconf.rb in the fastfilereader dir (or anywhere else in the ext dir where there might be more nested extconf.rb files.

then run this:

cd /usr/local/lib/ruby/gem/1.8/gem/eventmachine-0.12.11; rake gem:install

this should solve the issue of getting errors like this:

make: I.: Command not found
make: dynamic: Command not found

The Web needs a language

2010-02-25T17:42:00.000-08:00

I am now convinced that the Web needs a language of its own. It is not good enough to try and build systems with general purpose computer languages such as Java, C#, Python, or Ruby. Tick, tock, request, response, the collage of technologies that it takes to put together a simple website and backend is a wonderful display of inefficiency. I have seen the future and it is not hacking Javascript or HTML! How many ways can the same Rails app be written doing almost the same thing? Wouldn't it be nice to develop a solution to a problem using one language to express business rules? It seems like the best way to solve this problem is with better tools. New languages are judged on their expressive power, both the computer and human type. Most of the people using the Ruby language, do so because it will be used to generate what is between a couple of html tags.

I hereby propose that a new language be created that inherently knows the semantics of the web leaves us problem solvers with the ability to apply business log simply and directly.

PHP -- much better than using switch statements

2010-01-06T08:18:00.000-08:00

PHP's syntax is manic muddled mess. It is not object oriented programming, but there are a few things that make it easy to do things better than the average hacker.

Function pointers, if you are a C person, Funs, if you are an Erlang cat, Procs or lambdas for Ruby, delegates in C#, lambdas in my beloved Python is what I am talking about.

PHP too, supports this, but in a completely straight forward way.

take the following PHP code:

function iMRichBitch() { compute48thMersennePrime(); }

you can simply refer to pass around a reference (or pointer) to 'iMRichBitch' using a string by the same name. Of course, when the function is to be invoked, you simply can say:

$func = 'iMRichBitch';
$func();

So this begs the question, "if I have function pointers", then I ought to use those instead of case statements and maybe even nested conditionals, right?" The answer is a big 'YES'.

so instead of this:

switch($conditional) { case 1: do1(); break; case 2: do2(); break; default: break; }

you might think about doing this (with a little defensive code for garnishment):

$conditional();

Lambda Calculus

2010-01-03T14:35:00.000-08:00

Can anyone fully understand lambda calculus? If so, what would they do with that power?

Functional Programming

2009-10-08T15:05:00.000-07:00

Your brain is rolling along on that cold steel track of life. You process item after item, then list after list as if you were the building the chassis for a million robotic arms. Do this, then that, then this. Wake up, shower, go to work, go to lunch, come home, go to bed, rinse and repeat. Suddenly div/0, your brain stops and you forget every pattern that was pitted into your mind. The very wiring of the mind is a melted cacophony of cold snow and bloodied bird feathers. You know not who you are or what you are or why you are. Objects become functions and anonymous functions become parameters. Logic is modeled in actors passing messages. Actors mean nothing beyond what messages they pass. Cars on the highway become mail to mailboxes. Action and nonaction and not being. Colors become transformations yet to be decoded. You are what you do and nothing more. That is the primordial essence of the functional being.

Reading list

2009-10-05T05:31:00.000-07:00

I am logging a reading list for myself. Most times, when reading something, the author references another work, so I want to log this list as a must read.

Joe Armstrong's doctoral thesis
Why do computers stop and what can be done about it. by Jim Gray

Perl one liner, backreferences, displaying only part of a line

2009-09-30T13:54:00.000-07:00

I like this construct for searching through files in a directory and pulling out part of a line.

Just add the regular expression. The '?' designates a non-greedy expression, which means the finite statement machine will quit on the first successful match.

find . -name "*.xml" | xargs perl -ne '/(.*?)"/; print $1; print "\n";'

What I like about living in Atlanta, GA

2009-09-29T06:03:00.000-07:00

I recently moved to the Atlanta area. @dneighbors on twitter asked the question "Why Do You Live Where You Live?" It may be that I just love Atlanta, or maybe cause it felt so good to get out of Phoenix. Let me paint a picture for you.

Here is my perspective. Being born and raised in Phoenix, AZ and then moving to the Atlanta area in my thirties was a monumental change. The culture, weather, terrain are all starkly different.

Phoenix: hot, dry, dusty, smoggy, transient culture, California rejects, kind-hearted souls, sunburn, sunny days, prickly cacti, rocky moon-like surfaces, smelly dairy farms, cotton fields, traffic cameras, migrants standing on corners, 90 degree low temperatures, blast oven heat in your face from April to November, the sweet smell of jasmine and orange blossoms on spring nights.

Atlanta: Cool breezes, smell of pine in the air, festivals in the park on beautiful days, a sense of community, long summer days by the pool, peach cobbler, wild rasberries, blackberries, and blueberries growing in summer, smell of chocolate in the spring when the Magnolias bloom, blooming wisteria, azelia flowers, the old narrow streets of mid town, the rolling hills of Atlanta, every street named "peachtree" downtown, the aquarium, the college football, the reddest red of the maples in the fall, the greeniest of green of the pines against the deep blue sky, the "oceans" of kudzu, the brisk autumn breezes delivering a blanket of red, brown, and yellow leaves, the southern hospitality, neighbors helping neighbors, the squirrels, the bright red cardinals, the shady paths in the park, the fireflies, the geese, the deer

Out of work

2009-09-21T07:23:00.000-07:00

I recently lost my job. I am looking for a new way to earn a living. I have a couple of leads right now and I am also reading a lot and hacking on some rails apps for some of the business ideas that I have had over the last few years. It feels good to get out of on my own. I am learning about some things that I have wanted to know about for quite some time. Some Erlang topics are really interesting me right now. It would be really nice to start a consulting firm. I have a feeling that jumping into a long term full time job is not the cards right now.

I had an idea for a documentary. It would follow a small software engineering firm from inception to consulting and producing a piece of software to trying to paid. All the things that small businesses have to deal with. It would cronical the times we live in. Our culture here is very enterprising and this where creative projects go from a concept to a real living thing. I dream that I could be the one who starts the company and filmmakers can just hang around and record everything. I would love to watch that kind of a movie. I was fascinated with the documentary, "The Staircase". I highly recommend this film. These are the best sort of movies, I think.

DNSMASQ and multiple DHCP networks with DHCP relay

2009-08-14T07:55:00.000-07:00

I switched from MaraDNS to DNSMasq. DNSmasq is an excellent tool for small networks. Small networks are defined as something less than 1000 hosts. This is most networks! I love DNSMasq's simplicity. I also like how it bundles DHCP, DNS recursive server, DNS caching. The DHCP is surprisingly simple to use and configure and incredibly feature-full. For example, the DHCP server handles PXE clients requests for the next server. It also handles requests that are relayed, so you can provide multiple subnets DHCP service and do it based on the tag provided by the relayer. I also like how you can reuse the local (to the DNSmasq) machine's /etc/hosts and /etc/resolv.conf to gather static entries and upstream recursive DNS caches. It seems simple to just reuse these files for the server and clients. You are essentially exporting the server's /etc/hosts and /etc/resolv.conf to your clients.

Tail recursion in Python

2009-05-29T06:52:00.000-07:00

Problem: When marshalling an object to be able to transmit to or from an XMLRPC server and the xmlrpc lib is insufficient to the task because the objects within the main object do not implement a marshal method.

Admittedly the datetime, mxDateTime and None objects should just know what to do if asked by the xmlrpclib to marshal themselves, but in this case, I want to do this myself.

Answer: Here is a tail recursive way to perform this task. This should not use any stack space, but I cannot confirm this since I am not sure what is going on in the interpreter.

def __convertNonMarshalables(self, obj):
  if type(obj) == type({}):
    for key, value in obj.items():
       if value is None:
        obj[key] = ""
      elif type(value) is type(mx.DateTime.now()) or type(value) is type(datetime.datetime.now()):
        obj[key] = value.strftime('%Y-%m-%d')
      elif type(value) == type([]) or type(value) == type({}):
        value = self.__convertNonMarshalables(value)
  elif type(obj) == type([]):
    for value in obj:
      if value is None:
        value = ""
      elif type(value) is type(mx.DateTime.now()) or type(value) is type(datetime.datetime.now()):
        value = value.strftime('%Y-%m-%d')
      elif type(value) == type([]) or type(value) == type({}):
        value = self.__convertNonMarshalables(value)
  else:
    if obj is None:
      obj = ""
  return obj

Leopard killed TextMate, sort of

2009-05-26T14:14:00.000-07:00

Just updated Leopard and now having "#!/bin/sh" in the "command(s)" section of the bundle editor for any bundle yields "bad interpreter: no such file or directory" if I remove the directive, it works fine, but now on the shell bundle, the run command script is a ruby script and TextMate will not take #!/usr/bin/env ruby, yields bad interpreter again, but I really need to tell TextMate this is a ruby script. So I ran the script with ruby -e'do something'. That is weak sauce but it works. I also noticed circular dependencies in the bundles. You need ruby to run a python script in the python bundle; you need bash to run a ruby script, you need ruby to run a shell script.

I am not sure why this is because the text in command(s) should be treated as a shell script. Anyway, if anyone else has this problem, then removing the shell directive line should help.

Functionalness

2009-03-10T18:20:00.000-07:00

An object that has no function does not exist.

Auto adding DHCP host names to DNS (MaraDNS)

2009-03-07T14:58:00.000-08:00

Ever wonder why a host that you offer a DHCP lease to just does not automagicly appear in your non recursive DNS server's zone file. Well, it should! Using this "glue" code, it will be a snap to update your zone files whenever a new host joins the network. In this example, I am using MaraDNS and dhcpd.

Overall concept:

dhcpd --> lease file --> cron script --> ruby script --> DNS zone file

Cron entry (/etc/cron.d/update_dns):

*/5 * * * * root sh /usr/local/bin/parse_leases_add_to_dns.sh

parse_leases_add_to_dns.sh:

#!/bin/bash

#check modification time
if [ ! -f /tmp/leases_mod_time ]; then
touch /tmp/leases_mod_time
fi

#see if the last check time is less than mod time
if [ /tmp/leases_mod_time -ot /var/lib/dhcp/dhcpd.leases ]; then
/usr/local/bin/ruby add_hosts.rb
touch /tmp/leases_mod_time
fi

add_hosts.rb:

leases = open('/var/lib/dhcp/dhcpd.leases')
dns = open('/etc/maradns/kleetus.1337.foo', 'w')

leases_string = leases.read
lease = ""
client_hostname = ""

leases_string.each do |line|
next if line[0..0]=="#"
unless (line =~ /lease/).nil?
lease = line.split[1].gsub("\"","").gsub(";","")
end
unless (line =~ /client-hostname/).nil?
client_hostname = line.split[1].gsub("\"","").gsub(";","")
end
if client_hostname!="" and lease!=""
w_string = "#{client_hostname}.% #{lease}"
dns.write("#{w_string}\n")
lease = ""
client_hostname = ""
end
end

statics = ["myth-test.% 10.20.10.120", "*.% 10.20.10.100"]

statics.each do |line|
dns.write("#{line}\n")
end

dns.close

load=`/etc/init.d/maradns force-reload`

So why are you using a ruby script, can't you just do it all with bash/awk?

Sure, but the ruby script is more fun to write and maintain; if you feel the need to use bash, then this is possible too.

Adam Carolla

2009-02-23T06:30:00.000-08:00

If you have never heard of this man, you are for a treat, my friend! Adam is the Quintessential everyman's man. He can be heard here

Here are the reasons why I love Adam Carolla:

1. He connects with his listeners on a personal level because he doesn't act like some stuck up rich person. He recognizes that most of his listeners are just like he is. He worked construction and cleaned carpet. Now he has means, so it is nice that he can create a portal into the world of Hollywood, but he always remains genuine. He is sincerely connected to "the rest of us".

2. He is honest. This really should be number 1 on my list. You cannot connect with people if you are phony. People can see through this in a second and radio people are fooling themselves if they think that they put on a persona and retain an audience. Ace Man (Carolla) just puts himself out there, take it or leave it. He has flaws and those flaws can be endearing since we all have them too.

3. He is observant. This is more than schtick, Adam brings out the truly absurd in our culture. He comes off more genuine that the Seinfeld ("Who are these people") bits. He gets into more counter-culture routines that are really funny. He has one bit where is mimes a strip club MC. Callers will ask him to do an impromptu routine centered around them. Great bit.

4. Connections. Adam is plugged into Hollywood and its weirdness. He will call out weird shit too. Life is all about relationships. Adam's personality is likeable and most pop culture stars like Adam, so he can leverage this.

5. Adam is old school So. California. He was born in North Hollywood and lived amoung the regional culture. I find people who talk about subculture amoung the most interesting. I live in Atlanta, so this is far removed from this, but I love to hear about how "the Valley" is like or dislike my area.

How to get ffmpeg to transcode your over the air mpeg2ts to iphone 3G

2009-02-20T14:32:00.000-08:00

I got this figured out, so I thought I would share this with anyone transcoding mpeg2ts (over the air high definition) to something that the iphone 3G can play from a streaming server such as apache. Here is my ffmpeg line:

ffmpeg -i "${directory}/${file}" -async 3 -r 29.97 -acodec libfaac -ar ${rate}
-ab ${abitrate} -ac 2 -s ${width}x${height} -vcodec mpeg4 -b ${vbitrate} -flags
+aic+mv4 -mbd 2 -cmp 2 -subcmp 2 -g 250 -maxrate 512k -bufsize 2M -title "${file}"
 "${directory}/${file}.mp4"

The key here is to ensure the playback is at no more than 30 frames a second, or your iphone will play like 1 second and then stop...like it can't buffer the stream. Of course, when you do this, you will lose audio/video sync since you just yanked some frames out out the stream, so you need to use -async 3 to fix this; 3 works for me, but you may want to experiment. Also, I am using a recent SVN trunk compilation of ffmpeg with libfaac and xvid enabled in the compilation. This made a huge difference in whether ffmpeg could work with me on this TS file.

Now the bash like variables are exactly that, bash variable interpolations. Rate is the sample frequency of the audio...something like 44100 or 48000, abirate is the audio bit rate...something like 128k, the vbirate is not crucial, the video bit rate, something like 480K or 378K. The vbirate will really make the resulting file smallish. The original was prolly 2 or 3 times that. The dimensions usually work out to be 320x240, but I have also used 432x240 with success for 1.78 ratios.

Ipsecuritas, Mac OS X, Sonicwall Enhanced Firmware

2009-02-20T13:23:00.000-08:00

This was a tricky one. I always had some difficulty getting the free IPsecuritas connected; so I am sure others had problems too.

Using IPsecuritas 3.2 build 2501
MacBook Leopard

connecting to a:

TZ 190 Wireless Enhanced
SonicOS Enhanced 4.0.1.3-46e

Sonicwall side config (straight out of the tech support report)
--- SA 1 ---
Authentication Method : IKE with Preshared secret
VPN Policy Name : "WAN GroupVPN"; enabled
Policy Type : Client Policy
Pre-shared Key len : 14, value=
IKE Local Id : UNKNOWN
IKE Remote Id : ID_FQDN: (GroupVPN)
Local network :
Peer network :
IKE Exchange : Aggressive Mode
IKE Proposal : DH Group 2; Encrypt/Auth - 3DES/SHA1
IKE SA Life time : 28800 (seconds)
IPsec Proposal : DH Group 2; Encrypt/Auth - ESP: 3DES/HMAC SHA1
Ipsec SA Life time : 28800 (seconds)
Policy Options : PFS: on; Xauth: on; Netbios: on; Multicast: off
Management : HTTP: n; HTTPS: n; SSH: n
XAUTH user group : Trusted Users
Default LAN gateway : (0.0.0.0)
VPN policy : Bound to zone WAN

WAN GroupVPN Client Settings:
User Name and Password Caching:
XAUTH User Authentication is Required
Cache XAUTH User Name and Password on Client: Never
Client Connections:
Virtual Adapter Settings: DHCP Lease or Manual Configuration
Allow Connections to Split Tunnels
Set Default Route as this Gateway is Not Selected
Apply VPN Access Control List is Not Selected
Personal Firewall on Client Machine is Not Required
Client Initial Provisioning:
Use Default Key for Simple Client Provisioning is Selected
-----------------------------

Now the ipsecuritas config

General Tab:
--------------------
Remote IPSec Device: IP or host name of Sonicwall (must be reachable from Internet)
Endpoint Mode: Host (IP Address left blank)
Remote Mode: Network (Internal LAN network of the Sonicwall, such as 10.0.1.0 CIDR/Mask 24)

Phase 1:
--------------------
Lifetime: 8 hours
DH Group: 1024 (2)
Encryption: 3DES
Authentication: SHA-1
Exchange Mode: Aggressive
Proposal Check: Claim
Nonce Size: 16

Phase 2:
--------------------
Lifetime: 8 hours
PFS Group: 1024 (2)
Encryption: 3DES
Authenication: HMAC SHA-1

ID:
--------------------
Local Identifier: Address
Remote Identifier: FQDN.... just fill in the "Unique Firewall Identifier" from the Sonicwall VPN section
Authentication Method: XAuth PSK
Preshared Key:
Username: XAuth username
Store Password: checked if you would like the password to be stored
DNS: check "enable domain specific DNS servers"
Domains: fill in your domain name
Name Server Addresses: probably your domain controller ip address

Options
--------------------

Check off the following:
IPSec DOI
SIT_IDENTITY_ONLY
Initial Contact
Support Proxy
Request Certificate
Send Certificate
Unique SAs
IKE Fragmentation

NAT-T disable
do not check "enable connection check"
Action after connection timeout= Give up

--------------------

The key for me was Perfect Forward Secrecy was NOT enabled but it should have been! So ENABLE perfect forward secrecy. The reason for this was that IPSecuritas just does PFS without an option to turn it off or on, so you must turn it on, on the Sonicwall. Otherwise you will get "NO PROPOSAL WAS CHOSEN" when trying to negotiate phase 1. Always have your log file open when trying to debug these connections. Also, be wary of mapping multiple networks behind the Sonicwall, each has to build its own contract. Please contact me if you need help with your connection.

UAW and the first qid pro quo for the Obama Administration

2008-12-13T06:46:00.000-08:00

Let's call this bail out what it really is and stop insulting the electorate's intelligence. This is the first payment to the most powerful of Obama's supporters. The UAW had the unmitigated gall to send Obama a bill for services rendered. So now, we all get to sink billions into car companies that have not been competitive since the 1960's. Kleetus' message to the Big Three. You lost! You were beaten! Have some diginity and go do something else! But neigh, what are the car companies and their allies doing? If we can't beat Toyota and Honda by making cars people actually want to buy, we will just finance politicans' campaigns and send them a bill afterwards to get free money. We can spin this thing saying that "wouldn't it be terrible if millions that we employ lost their jobs"? Guess what? After you receive the bail out money, and you will, you will STILL go out of business, you dopes! If you couldn't build cars that people wanted to buy before in a good economy, what makes you think that you can suddenly be competitive with other car makers in this economy? Who is going to buy your cars now that you are on the brink of going under? Do you think that we will keep printing money for you all indefinitely? I would think not.
Let's just cut to the chase on this bail out and call it what it is; it is taking money from hard-working people in places not Michigan and sending it to Michigan and places under the thumb of the UAW. This is government highway robbery. This is the cost of the electorate not paying attention. This collective cost of not knowing sh** about your government is that "Tony Soprano" types in the unions and inept CEO's in Detroit get YOUR money. I think when Obama said "Change you can believe in", he was speaking DIRECTLY TO DETROIT! Translated less muddled, "Believe me guys, I will be sending you change, bags and bags of it". These guys are thieves and they have Obama in their pockets. Really they have the House and Senate leadership since Obama really has no power and will only be a puppet for the unions and trial lawyers. Maybe we ought to petition these groups for the next 4 years?

What is with that title?

2008-12-13T06:38:00.001-08:00

Logic programming is the culmination of mathematics and its study of natural patterns. Prolog functional programming is the language in which to express these patterns and process them in the most powerfull mechanical computing machine possible; The Turing Machine. Rather, a physical machine that closely resembles Alan Turing's machine. The title of this blog represents a rule or a statement evaluating to true that can be processed unambiguously by a machine of my choosing. The things written here aim to be unambiguous and truthful.

Cell Annoyances: Simple, Clear, Direct, and Concise

2008-10-08T05:10:00.000-07:00

Found this article "Peering Inside a Mobile Phone Network" written by Rich Mogull. It was very well written and cleared up some things that I did not know about the cell networks. I had the grave misconception that the cell networks are like IP networks; oh no sir. The truth is that they are much different. The article sort of takes the view that we (users of these networks) should think about cutting the operators and designers of these networks some slack. I say no way on that. Not our (users of these networks) fault that the cell networks are woefully inadequate to the task. Alas, the article described (from a very high level) what was taking place and it is worth a read even if you are very technical already.

Python Code:: Sort by Key and Subkey in a List of Dictionaries

2008-09-18T06:42:00.000-07:00

Here is a semi clear way to perform this function. There is probably a more elegant way to do this or even some snazzy library method, but this shows some of the guts of how this might be done. I am confident this can be rewritten to perform this is linear time or better. Again this is quick and dirty, like "Ah crap, I need this now and I can optimize later" type of thing:

def sortResults(self, unsort_rs, orderby, desc, subsortkey=None):
   list_rs = list(unsort_rs)
   result_rs = sorted(list_rs, key=itemgetter(orderby))
   if subsortkey:
     sub_list, complete_list = [], []
     for i in xrange(len(result_rs)):
       sub_list.append(result_rs[i])
       try:
          if result_rs[i+1][orderby] != result_rs[i][orderby]:
            complete_list += sorted(sub_list, key=itemgetter(subsortkey))
            sub_list = []
        except IndexError:
            complete_list += sorted(sub_list, key=itemgetter(subsortkey))
     result_rs = complete_list
   if desc == 1:
     result_rs.reverse()
   sorted_rs = tuple(result_rs)
   return sorted_rs

Ipsecuritas, must allow ICMP, MAC OS X VPN

2008-08-20T07:48:00.000-07:00

So we had this box nonstop pinging a server. This IP was private, so it was easy to tell it was from the VPN zone of the firewall, but I could not tell why someone was doing this. So I filtered this out and waited for the calls to come in. Well, the call came in about someone getting time outs from this server when pulling web traffic across the VPN. Surely, web traffic has nothing to do with ICMP that I filtered. Well, I would be WRONG. It turns out that Ipsecuritas VPN client uses a nonstop ping (once every 3 seconds) to a LAN host that it previously had traffic to in order to keep its tunnel open. Otherwise, IPsecuritas (client side) will tear the tunnel down. Game over for the VPN connection. Bunk! Shame on you Ipsecuritas (or Apple), are you so ghetto that you need to do this? It just seems so bush league. From a sysadmin point of view, you can't think of a better way to do this? How about a proper keep alive packet to the firewall?

Sharing stuff

2008-08-14T06:17:00.000-07:00

Ok, how can I be of service to someone else? How can I enrich someone else life? How can I ease the suffering of others? These are the questions that I ask in my life. In pursuit of this, I think about experiences that I have had. For some reason, I think it noble to impart any scraps of wisdom that I have picked up along the way. Maybe, I can impart wisdom in the fashion of, "hey I did this and it hurt, so be smart and don't do it, at least not like I did".

With all this in mind, I am going start a series of "Life on nuclear submarine", so that people can get a sense of the tribulations being a crew member. This was a time when I was constantly undergoing strife. Maybe lessons can be taught about not being a sailor, or maybe there are just lessons about being in close proximity to 100 other men.

Bikes for Christmas

2008-08-13T08:08:00.000-07:00

So I started this charity project called "www.bikesforchristmas.org". I totally forgot to explain what that was on this forum. Maybe some marketing value will come my way. Here is the thing: It is just me giving bikes away for Christmas. That's it. You got a youngster that needs a new bike, just sign up and I will provide.

Get a bike!

Bacula: Understanding its Pool Resources

2008-08-03T15:00:00.001-07:00

I back up about 20 different data sets daily. Each set exists on a separate machine somewhere on our network. I struggled a bit to get Bacula to do what I wanted it to do. Maybe I was looking at the problem wrong, but I felt as though I should formulate my own back up strategies and work with Bacula's configuration to devise the tactics. It seems that you should strategize around Bacula's common usage to make it easier on yourself. Bunk! Bacula does the job, but you need to understand the configuration settings well to get Bacula to do what you really want it to.

First, the pool resource section of the bacula director configuration is a good place to start. This is the stuff I monkey with the most.

My strategy (not thinking about how Bacula does things). I want the following:

1. on the first Monday of the month I want a FULL backup of all data sets.
2. on the 3rd Monday, I want a differential backup everywhere
3. every night other than those two, I want incrementals done.
4. I want a pool for each machine (each data set).
5. I want a maximum of 2 volumes written for each data set
6. the data set will be a month of backups (FULL backup, incrementals, differential, incrementals)
7. once 2 volumes are written out in the above nature, I want the oldest volume recycled. This gives us 2 full months of backups, at best and 1 full month backup at worst, depending on where we are in the backup cycle.

To make this happen, you must do the following:

1. set your "Maximum Volumes" to 2 for all the pools (in pools resource section of bacula-dir.conf)
2. set your "Volume Use Duration" to 1 month.
3. set your "RecycleOldestVolume" to yes
4. set your "Recycle" to yes
5. set your "Purge Oldest Volume" to no
6. set your "AutoPrune" to yes
7. I recommend letting bacula auto-name your Volumes. I mean the pool has a descriptive name, so who cares what the volume is named? So set "LabelFormat" to "vol"

Then g'head and fire off your backups for a month. Then come back to the config and change one setting once your pools each have 2 volumes written in them. In this case 32 days after starting your backups. In bacula-dir.conf change the setting for "Purge Oldest Volume" to yes by doing the following:

1. edit your bacula-dir.conf file for each pool and restart the bacula director.
2. update each Volume in the already written volumes to honor this by issuing a "update volume" at the bconsole and changing this parameter for each volume.

Now you have a self-rotation backup schedule that will maintain itself, more of less.

VirtualBox PXE Boot or VirtualBox PXE / TFTP

2008-07-31T08:13:00.000-07:00

Just started using Sun's VirtualBox. VMware, Parallels ought to shaking in their closed sourced boots. It was easier to install and configure than Parallels and VMware and is FREE (as in beer). Sun even released a open source version. Parallels is such a bane running on Mac OS X, but VirtualBox seems to run smoother. I was even able to PXE boot a guest OS without using a floppy image like I have to do with my older version of Parallels.

A couple of things make VirtualBox different than other hypervisors. First, at least on the mac version of VirutalBox (VB), the networking differs in the following ways. This is in the manual too.

1. The network between the host OS (the OS that is native to the machine and that VB is installed on) and the guest OS (the OS that you are installing inside VB) is implemented in user mode. What this means is that VB does not reach into the host OS kernel and try to glob onto the network stack, Instead, VB just takes network traffic from its guest OS's and sends it the same way any other app on the host does. VB decided to do this using NAT by default. So VB just needs to present one logical "port" to the host OS for the network. So you can think of the VB hypervisor as a router, just like the one you may use at home to have multiple computers behind a cable/DSL modem. So the hypervisor has the ip of the host OS and it masquerades all its guest OS's as this ip. Pretty easy, makes sense, but definitely less convenient.

2. You can "bridge" the network connections to your guest OS's too. This is more common in paravirualization and virtualization. There are directions to do this in the manual. There appears to be no way to do this for VB for Mac OS X (using VB on Mac OS X host OS). Oh well, maybe in later versions. You can still port forward while you use NAT. I would not need this feature at this time anyway.

3. I installed VB on Mac OS X host OS and then PXE booted the debian etch installer and noted some things that can save you some troubleshooting time. First, you must make a TFTP directory and copy your debian installer files and pxelinux.0 image there. Also, and this tripped me up good, you CANNOT pxe boot into an image that has a space in it. For example, I named my VB image "warehouse image". So when pxelinux tried to go and find its configuration (by default: pxelinux.cfg/), VB passed it a directory prefix of "warehouse" WITHOUT the 'image" part. Bad news. I renamed the image to just "warehouse" and it worked. So here is the list of what to do to make VB PXE boot.

a. create an image in VB (be sure not to use white space in image name, use foo and NOT foo bar)
b. configure the boot options to boot from network
c. create a new directory called TFTP in ~/Library/VirtualBox/
d. on your mac, rename "pxelinux.0" to foo.pxe and copy it to ~/Library/VirtualBox/TFTP/
e. copy the entire directories of pxelinux.cfg and debian-installer to ~/Library/VirutalBox/TFTP/
f. the pxe system can be obtained from debian in a netboot file called netboot.tar.gz (this contains all the files you need to pxe boot into a debian installer).

A couple of things I noted AFTER debian was installed and I reboot into it. I reboot after everything was installed and since it was set to network boot before, it tried to network boot again. This time, it got an IP from the internal VB dhcp server (yes, the hypervisor runs an internal dhcp server), it then found the tftp (next server) and boot attempted to find a configuration inside pxelinux.cfg but COULD NOT because "TFTP server does not support the tsize option". OK so I let it sit and about 10 minutes later, it booted anyway! I have seen this tsize error before and it is well documented in the pxelinux FAQ's. They recommend using a TFTP server that supports the tsize option such as "tftpd-hpa", but why the VB tftp server worked before, but not after the debian install is a mystery. Installing an OS should not have anything to do with this tftp server.

Ruby -- How to turn a csv file into a list of has

2008-06-24T06:56:00.000-07:00

There are convenience libraries for this type of thing, but I thought I would plunk down my solution.
Say you have a csv string and you need to just turn it into a proper Ruby data structure. A list of hashes would be nice since this is the same type of data structure you typically work with when you query a DBMS. Here is a function to do this.

def turn_csv_into_list_of_hashes(string)
  returned_list = []
  #the first line should be header
  rows = string.split("\n")
  header = rows.shift.split(',')
  rows.each do |row|
    row_hash = {}
    row.split(",").each_with_index { |item, i| row_hash[header[i]] = item }
    returned_list << row_hash
  end
  return returned_list
end

Erlang movie and other cultural turning points

2008-06-12T05:50:00.000-07:00

Not sure if you've seen it. The Erlang movie shatters all notions of what human beings can accomplish with a bit of film, some astounding acting, and a script that descended right out of heaven. There are two time periods in human evolution, pre and post Erlang movie.

How environment variables really work (in POSIX systems)

2008-06-03T04:48:00.000-07:00

Environment variables are strange animals, straddling the system (not kernel) and the application world. I will use the analogy from the movie "The Matrix". If you have not seen "The Matrix", stop reading this and run to a video procurement establishment and get it. In the Matrix, Morpheus shows Neo "the construct". This was a blank space or "environment" from which to load anything they needed. From huge racks of guns to grenades to cool leather jackets and sunglasses.
From that environment, they could have all the tools they needed to take on the agents. The supplies would always be there in the no matter where they went inside the matrix. This an excellent example of what environment variables really are. They are containers or "racks" that hold stuff or "information" so that actors (like the real actors) can use when inside the environment (matrix).
In POSIX systems like Linux, there are some rules that we must know about to know where and when environment variables are loaded.
Environment variables cannot just hang out without "being hosted" by another application. In Linux, there is ALWAYS a hierarchy of processes (applications, if you like). The Linux kernel is the mac daddy "process", but it is not a process per se because it runs entirely in a space that humans cannot access. I brought up the kernel as process because it really does run on the computer and launches the real first usable process called: init. Init is always process number 1 and ends up spawning all the other processes in the system such as X11, sshd, everything that runs in userspace. If you are a knowledgeable in Linux topics, you may have noticed some holes in the above, but for beginners, this explanation will give a decent primer.
Generally, applications in the Linux world know about the concept of environment variables and can make use of them, but the init process is different. It does not know about environment variables (because essentially there is no environment when it starts, except from the kernel). On the other hand init does know about arguments that are sent in before the kernel is booted. If you launch the kernel such as: vmlinuz foo=bar, the kernel will boot, examine the key foo, discover foo means nothing to the kernel and send foo=bar to the init process.
So init is special because it has no traditional environment. When init "spawns" its processes as dictated by init scripts or configuration in inittab, those processes will launch what is known as an interactive or non-interactive shell. This is where, I think, people get confused.

Now, the below information is sort of BASH concentric. This is the primary shell these days for Linux/Unix type systems. Lots and lots of people use other shells too, but if you are reading this, you probably are not interested in learning other shells other than BASH. The concepts for the other shells are surprisingly similar, but read slightly different files.

There are only two types of shells that you can have in a Unix-type system: interactive and non-interactive. The difference between interactive and non-interactive shells is that interactive shells involve a human or a process that needs the same things as a human would need when working inside a bash shell, for example. In other words, interactive shells require that something interact with the shell directly and not simply fork off and do its own thing.

Interactive shells:

To further confuse the issue, there are two types of interactive shells, login and non-login. Now pay attention, this is the good part. The files that interactive login shells read and interactive non-login shells read are DIFFERENT. This is why you should care to read this section. Let's look at the files that login shells read:

/etc/profile
~/.bash_profile
~/.bash_login
~/.profile

These files are read in that order. Other files can be read, but they will be referenced in the above files. Now long timers might exclaim:
"But what about if you run bash with the command 'sh' and use --norc option."
To this I say fooey. We aren't remaking the bash man page here.

For non-login shells (we are still interactive here) these files are read:

~/.bashrc

You might be asking when an interactive non-login shell might be used. Well, that is an excellent question. Non-login shells are used when you already are logged into a shell or even X and you need another interactive shell from which to launch scripts or issue commands. The system assumes that you do not need to reload /etc/profile or the above list because it loaded when you logged in! Now you might ask, so what if I changed something in my /etc/profile, but now I need the new variable and it is not there? This is because when you launched the shell, it was a interactive *non-login* shell. So now you are thinking "Christ, why so complicated?" No answer for that one, but you are singing to the choir, brother/sister. There are two things you can do for this.
1. The application you are starting may have an arg that allows a login shell startup....xterm does by launching, xterm -ls
2. Just throw your environment variables in an rc file such as ~/.bashrc and call it a day.
For number 2, I am sure shell purists are just ready to shoot me, but you know what? The whole thing is overly complex and kind of silly, so my theory is understand how the system works and then make it work the way you think it should and tell people why your system is better. If your way is not better, than people will fill you in as too why.

Non-interactive shells

A non-interactive shell would be for system "users", such as a web server, mail server, or a cron daemon. We like to put these users of the system into a shell environment where they can have just as much access to the system as they need to do their jobs and no more. They do not get to read /etc/profile because:
1. There is stuff in there about a human's environment (where the games are, possibly) and is of no consequence to them.
2. We do not want them knowing too much. If a cracker were to compromise that account. We do not want too much info there in that construct.
When you think of non-interactive shells, think of launching a shell script or running a PERL/Ruby/Python script from the shell. Also, you can think of your rc scripts that run when the system is booted. Those scripts still need an environment to work from, but should not be given the same environment as your bash prompt. You can also have non-interactive login and non-interactive non-login scripts. Wow this is confusing. The difference in the non-interactive context is that the login version simply looks to read:

/etc/profile
~/.profile

and the non-login version looks for a $BASH_ENV environment variable and attempts to source this file. The $BASH_ENV variable must have the full path because there is no $PATH in this environment, yet.

Bottom line for this post is that environment variables can be absolutely maddening. If you understand the constraints of the types of environments that you can have (interactive, non-interactive), then this goes a ways in figuring out where you variables will be loaded from and when. To help yourself keep all of this straight, I recommend one of two things.

1. make a cheat sheet for yourself.
2. memorize this info (or at least some of it) by taking an hour and experimenting with your shell.

The way shells and environments were laid out is very difficult to keep straight, but if you do not have the gumption to redesign it all, then I hope this post helps.

Rails Conf -- Impressions

2008-06-01T18:37:00.000-07:00

Really smart people, but unfortunately most were reinventing the wheel. Some knew they were creating things already available, but didn't care; others toiled needlessly. Obie's talk was clearly the best by pointing out that using your abstractions properly is clearly something to be valued.

Ruby VM's are not interesting. At least not to me. If you want a VM like the JVM, just use the JVM, you will be much happier in the end and maybe get to enjoy life more. Creating a faster/multi-threaded VM is a good learning experience, but does not mean much even in the short term.

Ruby does nothing for software safety than any other imperative computer codes. Although Ruby "makes programmers happy", this does not mean a hill of beans in improving our customers lives. If happy programmer == well-tested code that meets the specs, then great. But as Obie Fernandez points out, this is not frequently the case. Living the 80/20 rule through a world full of broken code stinks. I really like some of the research going on to allow Ruby to make applications that more concurrent, fault tolerant, and still be, well, Ruby. I hope some of these things make it into Rails Conf next year.

Rails Conf -- Joel Spolsky

2008-05-30T10:20:00.000-07:00

I really liked Joel's keynote at Rails Conf 2008. I have always read Joel on Software and thought he was a kool aid drinking Microsoft guy, but he did not come off as such. He was thoughtful and practiced. More later.

Apple "regedit"/ changing Terminal.app default settings

2008-05-17T09:00:00.000-07:00

There was not much on google for this. I recently clicked the "Use Settings as Defaults" in the "Window Settings" in Terminal.app for Mac OS X 10.4.11. This screwed me over because I was secure shelled into a remote machine at the time. What this did was save the last execution string in the Mac OS X defaults settings. So then every time I launched terminal, it would ssh into that last server! Bunk.

Well, I did not know about the whole "defaults" command in Mac OS X, I was looking for some config file in my home directory to no avail. Terminal.app is a system binary that squirrels away its settings in the internal os database. To fix this issue, you must access this internal database using the "defaults" process. Here is the exact syntax:

defaults read | less <------in less: hit '/' and type in: ssh -l, once you find the "section" label
defaults write com.Apple.Terminal ExecutionString ""

now we have knocked this problem out. This will also allow you to alter all kinds of things in your os.

How to take a bootable CDROM image and allow a PXE client to boot from it

2008-04-23T10:11:00.000-07:00

This took me awhile to figure this out. I had 30 clients that booted from a cdrom (they did not have an internal hard drive). They all ran the same cdrom image. When my company needed to expand, I thought it best to choose a thin client with no moving parts (this is a dusty dirty environment where the acidic dust wears bearings out!). The new thin clients have no hdd or cdrom, they must boot from the network. This cdrom image is proven over time and I wanted to continue to use it. Here is what I did:
Considerations: This is a linux/GNU environment, other OS are not considered.
1. Created a server to boot from (PXE server, DHCP server, NFS server)
2. Made a copy of the cdrom and mounted the copy, such as: # mount -t iso9660 /dev/cdrom /mnt/cdrom
3. Ensure /mnt/cdrom is readable by the world, such as: # chmod -R 754 /mnt/cdrom
TFTP/PXE setup
1. You will need to ensure that you have an initrd that will "do the right thing" when the linux kernel that boots from PXE. Here are the steps to manually recreate a initrd, this is not too hard:
a. Mount your initrd someplace, such as: # mount -o loop /boot/initrd /tmp/initrd
b. This image is read-only so you will need to copy it out and then edit it, such as: # tar cpvf initrd.tar /tmp/initrd && tar xvf initrd.tar /tmp/initrd_new
c. So now edit the 'linuxrc' or whatever your initrd uses as a 'init' script (the script that gets everything going).
d. when you edit this linuxrc, you will need to nfs mount the location that contains the files from the cdrom you mounted, such as: # nfsmount x.x.x.x:/mnt/cdrom
e. now create your new initrd image for use to pxe boot, such as: #mkfs.initrd initrd_new initrd
f. now put this initrd into where your pxe server expects it to be
FINALLY:
Reboot your computer into the same image that is tested.

Deleting all messages in an Exim4 queue

2008-04-17T07:27:00.000-07:00

All that is needed in this line is to substitute a pattern to find the messages you want:

mailq | sed -n '/\*\*\*/p' | awk '{print $3}' | xargs exim -Mrm

Python: How to find a list of keys for a given value in a dictionary

2008-03-26T08:26:00.001-07:00

Here is a straight-forward, easy to understand way to find a list of keys in a dictionary for a given value (that may or may not be in the dictionary)

def GetListOfKeysForGivenValueInDict(dictionary,value):
  keys = []
  for k,v in dictionary.items():
    if v == value:
      keys.append(k)
  return keys

usage:

dictionary = {1:2,3:4,5:4}
value = 4
keys_list = GetListOfKeysForGivenValueInDict(dictionary, value)
print keys_list

result:

[3,5]

How to write custom rules for spamassassin

2007-12-03T12:42:00.000-08:00

How to configure the spam filter for custom rules

do you really need to do this?

answer: if the customer is getting spam that is solicted or unsolicted and it is not marked spam and the customer is concerned, then yes. It is best to leave the filter alone until really needed because the kind of change you will be making will be global

* Ssh into your mail server and gain root privileges
* edit /etc/mail/spamassassin/local.cf
* add filters to the bottom of the file indicating your filter text
* here is an example that would work for marking email containing the phrase "foo bar is king" as spam:

body NO_FOO /foo bar is king/i

score NO_FOO 5.0

* save off local.cf
* restart the spam filter as such:

/etc/init.d/spampd restart (this depends on your implementation)

How to use rbot, svnwatch.rb plugin and post-commit to log svn commits to an IRC channel

2007-10-29T16:44:00.000-07:00

I have not seen this all in one place so here it is: (Obviously if you do not know what I am talking about here, then you do not need this)

How to setup svn (subversion) logging when a commit is made

1. using Debian, issue this command as root: apt-get install rbot
2. using other Linux'es, install the ruby gem, first search for it: sudo gem list --remote rbot
3. pick the rbot you want (get the latest stable) and install the gem
4. run rbot, the first time you run it, rbot will check to see if you have a ~/.rbot directory, if not, then the setup script runs
5. answer most questions as the defauit. The questions you need to fill in are the irc user, the irc server, the irc channel to join
6. ensure the irc channel to join is in the format: #our_cool_channel somepassword
7. once rbot loads and starts running, check to see if the rbot user you made up actually joined the irc channel. If it did, continue, of not, then go back and troubleshoot
8. Shutdown rbot by sending a CTRL-c to the term, then start it back up in "daemon" mode: rbot -b
9. Add "svnwatch.rb" to ~/.rbot/plugins/ This file should be on RobbyOnRails somewhere, search for it
10. Add the post-commit file somewhere (also on RobbyOnRails) where apache can run it when a commit is made, I recommend something like /usr/local/bin
11. create a symlink to the post-commit script in the hooks directory of each svn project you will to run this post-commit script for
12. For example, I want to have a script log to the irc channel when a commit is made to liberty sweet. So I would: ln -s /usr/local/bin/post-commit /home/svn/src/cool_project/hooks/post-commit
13. What this does is run the post-commit in /usr/local/bin/post-commit which fires off a commit message (on port 7666 by default) to rbot, which is logged into the channel.

How to create a reverse ssh tunnel

2007-09-17T14:55:00.000-07:00

Say you have a customer that has a firewall, but you need to ssh into their box and you can't mess with firewall (poking holes through it, etc.) If you can convince them to run a script you email to them or get to them someway, you can get a shell easily.

definitions:

local: (the customer's machine, which is behind the firewall)
remote: a box you have a shell on already

requirements:

both the local and remote computers must be running sshd (ports do not matter, but you will need root to listen on a port lower than 1024)

here is the line to execute on the local computer:

ssh -nNT -R 1100:local:22 remote

explanation:

you are ssh'ing from the local computer, past the firewall and into the computer you control. The 'n' option tells ssh to disregard stuff that comes in on stdin (we won't be sending local stdin over the wire, thus send it to /dev/null. The 'N' tells ssh to setup the tunnel but do not associate it with a command stream. 'T' means "do not give me a tty on the remote end". When the above command is run from a shell on the local computer, the shell will hang (after asking for login credentials) and no just a tunnel for other ssh sessions coming in from the other end. It will need to run during the entire session. Now from the remote computer you can simply ssh as such:

ssh localhost -p 1100 -l username

now you should have a shell on the local computer and have essentially defeated the firewall. For those of you reading this and wondering "Wow, someone can generate syn packets into my network, by tricking a user into executing some code, thus defeating my outer defenses.", you would be correct!

I always forget this, how to place a process in the background after it is already running

2007-06-03T21:27:00.001-07:00

So you started this process in your terminal and it is taking a really long time to complete. You want that terminal back but it seems too late to kill the process and rerun it with the '&' to detach it into the background. Well there is hope. You can complete the following to place the process in the background and get your terminal back.

1. From the terminal in question, the one where you do not have a prompt because the application is still running, hit Ctrl-z (this signals the process to pause, but not stop, it can be picked up again right where it left off)
2. Then type 'jobs', if you only have one job on that terminal (you most likely do), then this stopped job will be '1'.
3. Then type 'bg 1' or whatever the jobs output was. Blamo, you have restarted your process into the background.
update:
4. If you logout of the terminal (pty) which launched the process, you are essentially orphaning the process. This will cause the process to zombie (which means the process will never end/die because it is waiting on a signal from the parent that it received its result code) when the process finally does end. Also, if the process outputs to stdout/stderr, then there will be no pty parent associated with it, so Linux will cleanup this process.
5. Thanks to Charles Jones for pointing out that you can issue a 'disown %1' where 1 is the job number.

Reimplementation is back on!

2007-06-03T11:31:00.000-07:00

Exciting news over the past week. I have been given the opportunity to lead the IT department at the company I work for. This is good news because it gives me a chance to take the department in the direction I want (which includes updating the suite of tools.) I am excited about this because because Jade Meskill is finally free to fully devote his time to his own company, Integrum.

Looks like the reimplementation is off

2007-05-28T09:20:00.000-07:00

Looks like the company that employs me has decided not to upgrade the software they use. This is unfortunate. I will need to reevaluate how this book might get written. There will be no empirical data to include for the domain of the book.

socialDrinkr

2007-05-12T09:52:00.000-07:00

We just started on a new beer rating site. By we, I mean myself and 3 other people from work. Socialdrinkr is an experiment for those who love beer and would like to share their thoughts about the brews they love. We hope to make it useful for anyone. This is definitely a labor of love. All four of the developers have keen domain knowledge as well the knowledge to build the site. Alright time to get back hacking on socialdrinkr.

Writing a book!

2007-05-11T20:47:00.000-07:00

I have been given a great opportunity to write a book. My employer is giving me the opportunity to implement enterprise-level software using Ruby on Rails. I am very fortunate to be a part of this plan. Rails is new and exciting, but needs to be proven. We need to prove concepts like REST (representational state transfer) can work running a multiple million dollar operation. We need to implement a distributed database. Most importantly, we need Rails to work with legacy systems and data. This book will be a practicum of sorts, chronically the entire process from bidding the system to the client through to the "maintenance phase".

Going to Rails Conf!

2007-05-06T17:43:00.000-07:00

Excited = 'Yes'. I would really like to hear David A. Black speak as well as Dave Thomas (no not the Wendys guy), the Ruby guy. Myself and 6 others from Integrum will be going on May 16th. This should be a good time and a chance to show Integrum's colors in Planet Argon's backyard.

Living the dream

2007-04-05T08:06:00.000-07:00

Sipping a latte
Typing with one hand smartly
Igoring those around me

Coming at you like a spider monkey

2007-04-05T08:01:00.000-07:00

Greatest generation, my ass!