Real Linux SysAdmin

Ok, so you have a system with multiple SAN LUN's attached to it, and one of the logical volumes on that system is full and you need to grow it.  Well that would be easy if you still have free space in your volume group.  What if you don't?

Do you just add another LUN and grow your volume group, then extend your logical volumes?  Well you could do that, but what about next time, or the time after that, you don't really want a bunch of LUN's attached do you?  I can tell you that your SAN admin doesn't want to manage a butt load of LUNS on one machine.  So what do you do?

Well you expand an existing LUN! Thats what!

I know your thinking but how do I do that?  Well as it turns out it is really easy and I will try and convey the steps below.

1. Find the LUN number of one of the currently attached LUN's on your system, this will be the one we grow.
2. Have your SAN admin grow that LUN by whatever amount you need (100GB for example).
3. Reboot the server. (This is done because the system will still see the LUN as its original size even after the expansion until a reboot. There should be a way to rescan the scsi bus however everytime I try this it crashes my machines)
4. Confirm that fdisk sees the new size with 'fdisk -l /dev/<SAN DEVICE>'.
5. Run 'pvresize /dev/<SAN DEVICE Partition>', to change the physical volume metadata.
6. Run 'vgdisplay <vgname>' for the volume group that LUN belongs to and voila you should see the volume group now has free space, or more free space depending on your situation.
7. Simply run the normal "lvextend" commands to add space to your logical volumes.
8. Run the normal 'ext2online' command to see the new space on your logical volume, and bob's your uncle, you've got new space.

Now go and check your mount points and see all of your data still intact. Fantastic isn't it?

If you recieve this error and are trying to access something that has a self signed certificate, there are 2 places in Firefox that you must go and change in order to access this site again.

The first is under Tools--> options-->Advanced-->View Certificates

You must remove the certificates in the "Servers" tab, then under the same location you must remove the certificate authority because it is a self signed certificate, so on the "Authorities" tab, simply find the CA for the device you are having trouble with and delete it.

After these changes you should be able to access your device again without any problems, you will have to add another exception back to firefox, but that is the normal routine for self signed certs.

I hope this helps you.

Ok so if you have postgres databases and you get this error "FATAL: database is not accepting commands to avoid wraparound data loss in database "postgres""  The problem could simply be that you do not vacuum your database and it has reached a maximum row count.

You need to vacuum it in order to recover disk space and row counts after many deletes.  So how do you do that, you may ask, especially if your databases are "not accepting commands" .  Well as it turns out it is not that difficult, if you can still get into your database then simply login as a user that would have permissions to do this job and type:

sql>vacuum full analyze verbose 

And depending on how many rows are in your database the command can run anywhere from 20 seconds to 30 minutes or more.

If you cannot get into your database because of the above error things are a little more difficult but still not that hard, simply login either as postgres or root, and stop the postgresql server, then using the following command as postgres start the server in "backend" mode.

postgres -D /usr/local/pgsql/data other-options my_database

Provide the correct path to the database directory with -D, or make sure that the environment variable PGDATA is set. Also specify the name of the particular database you want to work in.

Normally, the stand-alone server treats newline as the command entry terminator; there is no intelligence about semicolons, as there is in psql. To continue a command across multiple lines, you must type backslash just before each newline except the last one.

To quit the session, type EOF (Control+D, usually). 

Note that the stand-alone server does not provide sophisticated line-editing features (no command history, for example).

Once you are in backend mode and have a backend> prompt you can issue the "vacuum full analyze verbose" command as above, then when it is finished then use Control+D to end your session. You should now be able to start postgresql as normal and then get into and issue normal queries to your database.

Sometimes the networking engineers that control your switches will ask you to "nail" your network interfaces to a certain speed and duplex.  They do this in order to be able to "nail" the switched end at a certain speed.

So if you need to temporarily "nail" your NIC, you can do the following.  First you want to check and see what speed you are currently at:
          ># ethtool eth0

This will tell you what you are currently at, it will look something like the following:

        Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: umbg
        Wake-on: d
        Current message level: 0x00000007 (7)
        Link detected: yes

After you know what you are currently at you can adjust it to what you want it to be at with the following command:
          ># ethtool -s eth0 speed 100 duplex full autoneg off

In this example I am setting the eth0 interface to 100, the duplex to full, and if you are going to "nail" a connection you don't want the auto negotiation to be on so we turn it off.

Now this is only temporary, when you reboot these settings will not persist, if you want these settings to come back after a reboot you must put the following lines in the file /etc/sysconfig/network-scripts/ifcfg-eth0 file:

           ETHTOOL_OPTS="speed 100 duplex full autoneg off"

This will cause the options you want to come back after a reboot.  There are basically 3 speeds, and 2 duplex settings that you could set for a network interface, these, for the speed, are 10, 100, or 1000.  When you check your current settings the interface will tell you what speeds it can do, your card may only do 10 and 100 and not 1000.

The duplex settings are either half or full and the explanations are below, (taken from http://www.tcpipguide.com )

Half-Duplex Operation

Technologies that employ half-duplex operation are capable of sending information in both directions between two nodes, but only one direction or the other can be utilized at a time. This is a fairly common mode of operation when there is only a single network medium (cable, radio frequency and so forth) between devices.

Full-Duplex Operation

In full-duplex operation, a connection between two devices is capable of sending data in both directions simultaneously. Full-duplex channels can be constructed either as a pair of simplex links (as described above) or using one channel designed to permit bidirectional simultaneous transmissions. A full-duplex link can only connect two devices, so many such links are required if multiple devices are to be connected together.

Anyway your doing this because your network engineer asked you to, so they need to know why and not you anyway.

So with these simple things in mind you should be able to set your network interface to any speed or duplex that your network folks want.

I hope this helps you in your daily admin lives.

This is a short that I did for the Fresh Ubuntu Podcast , I will probably do some more so keep looking back here and I will try and make some useful audio.  Thanks to everyone for their support.

If you need to sort some data sorted and you need it sorted on two different columns, like the test data below.

# cat list.txt

2007-10
2008-09
2007-12
2008-01
2008-10
2008-04
2008-05
2008-06
2008-07
2008-03
2008-08
2007-11
2008-11
2008-12
2008-02

Yeah simply dates by year-month, well I want it sorted by year and then month, so I would use sort like this:

cat list.txt | sort -t'-' -k1n -k2n

Which does a couple of things here, first it sets the dilimeter to a dash (-), so I don't have to worry about the dash and to put the data into two columns, the -k1n and -k2n tell sort to use the "key" in the first column which is a number, and then the "key" in the second column which is a number so you get out the following:

# cat list.txt |sort -t'-' -k1n -k2n

2007-10
2007-11
2007-12
2008-01
2008-02
2008-03
2008-04
2008-05
2008-06
2008-07
2008-08
2008-09
2008-10
2008-11
2008-12

Not that this was something so hard to do you couldn't figure it out, but it was something that I just used, and thought I would pass along about the "keys" (-k) option. Hope it helps someone out there.

Have you ever had a syslog server where you have a whole bunch of machines sending data to it, or a single device that is aggregating data from multiple sources before it sends to your device?

Well I do, I have a syslog server that recieves syslog feeds from about 100 machines, and I have a couple of devices that send aggregated feeds as well.  So I didn't want all of that data from multiples sources going into a single file. I wanted each host to have its own file.

rsyslog is a syslog service that can handle this quite nicely and easily. There are several ways to accomplish this task, rsyslog has properties that you can use for this. The properties can be found at rsyslog.com.  My simple task was accomplished in the following manner.

I edited the /etc/rsyslog.conf file to include the lines below to seperate into files by the "fromhost" and by a string match in the message.

The line to seperate by "fromhost" looks like this:
 
   # Server Logs
     # myhostname1 - Domestic Console
    :FROMHOST, startswith,"myhostname1"                       /var/log/servers/myhostname1.log
 
    # myhostname2 - Domestic Console
    :FROMHOST, startswith,"myhostname2"                       /var/log/servers/myhostname2.log

As you can see it is simply looking at the "fromhost"  field and using the "startswith" compare operation, I look for the hostname of the server and put it in a corresponding file.  Pretty simple huh?

As for splitting from a single source for multiple hosts it looks like this:

    :msg,contains,"hostname1"               /var/log/hostname1.log
    :msg,contains,"hostname2"               /var/log/hostname1.log
    :msg,contains,"hostname3"               /var/log/hostname1.log

As you can see this compare operator "contains"  is looking in the message itself and not a defined field as the "fromhost" did in the previous example.  This allows me to filter to seperate files by a string match, so it doesn't have to be the hostname, it could be an IP address, it could be by hour or month, or any other string in the message.

If you have machines that you have bonded 2 or more interfaces together you will see if you do an 'ifconfig' that both eth0 and eth1 have the same MAC as the bond.

bond0     Link encap:Ethernet  HWaddr 00:0F:1F:03:F8:E5 
          inet addr:10.x.x.x  Bcast:10.x.x.x  Mask:255.255.255.0
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:4703 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4344 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:410776 (401.1 KiB)  TX bytes:664401 (648.8 KiB)
 
eth0      Link encap:Ethernet  HWaddr 00:0F:1F:03:F8:E5 
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:4237 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4344 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:380928 (372.0 KiB)  TX bytes:664401 (648.8 KiB)
          Interrupt:185
 
eth1      Link encap:Ethernet  HWaddr 00:0F:1F:03:F8:E5 
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:466 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:29848 (29.1 KiB)  TX bytes:0 (0.0 b)
          Interrupt:193

In actuality both show the MAC address of the currently active slave.

So lets say that you are troubleshooting something and need to figure out the ACTUAL MAC addresses of the interfaces, well you have one via the method above, but how do you get the other one.

Well first I tried 'dmidecode' and there was no help there, then I tried 'ethtool' and no help there either.

So where to look, well I started thinking, everything is listed in /proc for the system, so I went poking around and found this. 

If you look into /proc/net/bonding  you should see a file that is the same name as your bond, in my case bond0, which contains the hidden gem you are looking for.

So simply cat out the file (in my case bond0) and you will get the following results.

Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005)
 
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
 
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0f:1f:03:f8:e5
 
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0f:1f:03:f8:e6

Now obviously your version of the bonding driver may vary, but you should see basically the same thing as above,and if you will notice the Permanent HW addr  lines in the sample above those will be the actual interface MAC addresses that you are looking for.

Well ok so I always said that Linux was stable and that you really "never" need to reboot, so here is my awesome uptime for one of my machines.

~]$ uptime 12:50:56 up 628 days, 21:10, 0.01, 0.02, 0.00

I have posted this here, because the CMOS battery will have to be replaced on this machine and I wanted to have record of this uptime.

This is not my longest uptime, but I didn't have a way to record the other online so we will start here.

There are several ways that you could get input from your users in a shell script. First of all they are all correct and depending on the situation you could use any one of them.

I am going to show you the "select" statement for the bash shell, and an external program called "dialog" later on. Both of  these will allow you to give your users a  menu like choice  and control  what  is  entered.

Ok so here we go, the select construct allows the easy generation of menus. It has almost the same syntax as the for command.

SYNTAX
      select name [in words ...]; do commands; done

The list of words following in is expanded, generating a list of items. The set of expanded words is printed on the standard error output stream, each preceded by a number.

So if you wanted to use this you might write something like this:

#!/bin/bash

echo "Which ice cream do you like best?"
select ANSWER in vanilla chocolate strawberry
    do
    echo "You chose $ANSWER"
    break
    done

This small example script would output the following:
Which ice cream do you like best?
1) vanilla
2) chocolate
3) strawberry
#?

The #? is where you put your answer.  If you choose 1 as your answer then you would get the following:
You chose vanilla
and the break causes the script to end the select statement, otherwise it would simply ask you over and over.

This is very helpful when asking for input from your users, this way you give them choices and you know exactly what the different answers are they could give.  If you simply asked the same question and did a read ANSWER on the next line you would get input from the user, but it might say red, and that wouldnt work very well in this case.