Linux: Find files greater than n size

Posted by & filed under Linux, Server Admin.

Recently I had a issue where I needed to clean up some disk utilization on a linux server. In order to find a list of larger files, I used the following find command from the root directory I want to recurse through:

As you can see, the -size switch is setting the minimum size to find at 50Mb. Another issue that I ran into was deleting a large amount of files at once using something like:

“Argument list too large” was the error. Apparently the list of files is too large for rm to handle. I found that there are a variety of methods to solve this, from using loops to split up the files into smaller groups, to recompiling the kernel. One of the simplest is to use the find command to delete the files it finds:

The list of files to get deleted can also be tuned so it does not delete all the files in the path:

[Windows] Finding a string in a bunch of files, then processing each of those files

Posted by & filed under Programming, Server Admin.

I had a task where there were thousands of files in a folder. Some of them contained a specific string and needed to be processed.

A quick n dirty method is to use grep (or windows grep in this case www.wingrep.com/) to identify the files and generate a list of filenames in plaintext. Then using a quick batch for…loop to process the files from the command line… nice and simple.

Substitute [process] for your command. %A contains the filepath from grep. In my case I wanted to just delete the file so I just replaced [command] with del. Done!

TechNet Reference: technet.microsoft.com/en-us/library/bb49…

Troubleshooting Cacti’s Poller

Posted by & filed under Server Admin.

Today I had a Cacti server just stop polling out of the blue. It happened around the same time a coworker added a device to monitor though I’m not sure if this has anything to do with it. The box needed a reboot anyway, so I went ahead and gave it a reboot to see if that would be a easy fix. Nope.

Disabled the newly added host, but still no dice. I checked the log file and found:

So the poller was running, but not actually polling… a ls-lah of the RRD directory shows the files have been updated, and have the correct permissions set. The graphs are seemingly being updated with null values.

Forced a run of the cacti poller (ran as sudo or I’d need to run as www-data):

Tail’d the log — note changing the setting under Settings -> General -> Poller Logging Level directly affects what gets logged here so it can he helpful to increase when troubleshooting.

Looks like it updated… not sure why it stopped. I sort of remember having to do this once before in the past and forcing the re-run fixed it then to IIRC.

Now that it did the full update… letting it poll to see if it does it again as it should…

UPDATE: It is still polling correctly — going to re-enable the host mentioned at the beginning and see what happens.

UPDATE 2: Everything is working good so far… increased my Poller threads from 1 to 2 and enabled process load balancing.

Pipe Viewer

Posted by & filed under BASH, Programming, Server Admin.

pv – Pipe Viewer – is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.

www.ivarch.com/programs/pv.shtml

Data Recovery: Finding VHD Files

Posted by & filed under Data Recovery, Server Admin.

SITUATION: Server had what appears to be a raid controller failure complete with it blowing away the data on the drives. The only backup available is corrupt.

BEGINNINGS
I began by mounting one of the drives on my local machine. Windows Disk Manager would not recognize the device as having a partition table. I then ran TestDisk. It was unable to recover a partition table. TestDisk also comes with a nice utility called PhotoRec. Photorec looks at the raw data on the device attempting to extract files. PhotoRec can bea bit of a pain when recovering a lot of files because it cannot recover filenames. Fortunately, this particular server was a virtual host, and the only files I cared about were the 10Gb+ VHD virtual disk files.

ADVENTURES WITH PhotoRec
I initially just ran PhotoRec just to see if and or what it could even find. After running for several hours, I was happy to see it found thousands of files. Upon closer inspection of the files I found that they were actually files that were inside the actual VHD’s. This makes sense because of how PhotoRec searches for files. I needed to create a custom PhotoRec file signature extension to handle the VHD files. www.cgsecurity.org/wiki/Add_your_own_ext…

Using a hex editor I opened several VHD files from my archive and noted thay all started with “conectix”. I then created photorec.sig in the PhotoRec directory with the following inside:

I then re-ran PhotoRec, this time using the options to only select my custom signature extension. It then began recovering VHD files. It found around 6 header signatures, and after some time had dumped them all out. I was able to mount one using diskmgmt.msc on my Windows7 laptop. The other VHDs were all unable to mount due to corruption or other factors.

Back to TestDisk
I decided to try TestDisk again, this time though when it asked what kind of partition, I selected none, then had it scan. It found the NTFS partition! I was able to navigate into the drive’s structure and instruct TestDisk to copy the VHD I needed to my local machine…

…After some hours, the dump froze. I was monitoring the file copy progress and it was making good progress (over 100G recovered) but then abruptly restarted. Or appeared to do so. Before I started this process I notices TestDisk was indicating that there were multiple VHD files with the same name. I am guessing it completed, then moved on the the next, overwriting the existing recovered file since no sane file system allows two files of the same name in the same directory. Eventually it froze as mentioned.

Data Dump
This should of been one if the first things I did, but I did it now since I wanted to simply attempt to repair/rebuild the NTFS partition table which should then give me direct access to the files rather than trying to extract them thru TestDisk. Obviously this should be the first thing done, but as I was confident in the physical drive’s health, I bypassed this at first.

I did the backup thru TestDisk but I believe it just used dd emulated thru cygwin anyway so it’s the same thing.

Rebuilding the NTFS Boot Sector On An NTFS Partition
I dumped the MBR and sadly it was all zeros thanks to the RAID controller nuking it.

Both NTFS boot sectors (the primary and the backup) are corrupted so we need to rebuild the NTFS boot sector.

TestDisk searches the MFT (Master File Table: $MFT) and its backup ($MFTMirr). It reads the MFT record size, it computes the cluster size, and it reads the size of the Index Allocation Entry in the root directory index. Using all these values, TestDisk can provide a new boot sector.

Finally it lets the user list the files before writing.

Update: That failed. The sector count is different now too. I suspect the data got hosed. I can easily recover with my dd command:

The Solution
Loaded up a copy of my dd into TestDisk, Created a new log file and selected the drive. For partition type I selected “None”. Then select Analyse. It located the NTFS partition and the I was able to hit (f or p — not sure typing this from memory) to recover files. I attempted this exact method previously, but tried to restore the entire directory. This time I simply had it restore the individual VHD I was interested in. Set the destination path (make sure there is space!!) and a few hours later I had a VHD… working flawlessly.

11-17-2012 Note: This was mentioned by the CG Security authors here: www.cgsecurity.org/mw/index.php?title=In…

gnokii — phone interface

Posted by & filed under Server Admin.

gnokii provides tools and a user space driver for use with mobile phones under various operating systems (most testing is done under Linux but also Solaris, *BSD family and MS Windows families are known to work). gnokii allows you to communicate with the phone over the serial cable connection, usb connection (support depends mostly on the operating system level support), infrared connection and bluetooth connection.

You can send SMS, receive them and save them in the phone, read and write to the phonebook, initiate and answer calls and more. Pretty cool and just what I need for a a GSM alerting project at work.

www.gnokii.org

Mosh: MObile SHell

Posted by & filed under Networking, Server Admin.

Remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes.

Mosh is a replacement for SSH. It’s more robust and responsive, especially over Wi-Fi, cellular, and long-distance links.

Mosh is free software, available for GNU/Linux, FreeBSD, and Mac OS X.

mosh.mit.edu/

Installing git on a cPanel server

Posted by & filed under Server Admin.

I needed to install git on a cPanel server recently. After adding the appropriate EPEL5 or EPEL6 repo (fedoraproject.org/wiki/EPEL), you should be able to simply do a:

But yum kept reporting a unmet dependency — a Perl-git package — even though I verified the missing package is actually present in the EPEL repo. After a bit of digging, I found cPanel has set yum to exclude any packages with Perl in the name. Simple enough to fix, but aggravating:

Remove “Perl*” from the exclude line and save.

Jump back into the yum.conf file and add the perl* exclusion back in so yum does not eat cPanel’s braiiiinnns….

Done!

16 Ultimate SSH hacks

Posted by & filed under Server Admin.

So you think you know OpenSSH inside and out? Test your chops against this hit parade of 16 expert tips and tricks, from identifying monkey-in-the-middle attacks to road warrior security to attaching remote screen sessions. Follow the countdown to the all-time best OpenSSH command!

www.itworld.com/print/261500

Ubuntu, Apache2 and relaying mail thru an external relay

Posted by & filed under Server Admin.

I have a fresh Ubuntu 11 server installation with the LAMP stack installed. When I sent e-mail thru PHP, the message never left the server.

I believe there is a more kosher way to do this, but this is what worked for me.
=> Modify /etc/mail/sendmail.cf
=> Locate the lines that say:

# “Smart” relay host (may be null)
DS

=> Edit the DS line like so:
DSsmtp.my.mailserver.com

Restart the services… good to go.

Visualizing Device Utilization

Posted by & filed under Networking, Server Admin.

Brendan Gregg recently posted some interesting data about visualizing large data sets. Particularly, device utilization which is a key metric for performance analysis and capacity planning. In his post, he illustrates different ways to visualize device utilization across multiple devices, and how that utilization is changing over time. The study included over 5,000 virtual CPU nodes and over 600 physical nodes on a production cloud environment.

Data visualization

dtrace.org/blogs/brendan/2011/12/18/visu…

Xymon 4.3.4 info column bug

Posted by & filed under Server Admin.

I recently deployed a fresh installation of Xymon 4.3.4. There were a few glitches which were easily solved by looking at the xymon logs. With one exception — the info column shows a “Internal Server Error”. Thanks to the Xymon mailing list, I found that there is a known bug in svcstatus.cgi in Xymon 4.3.4. The solution is to edit the source and recompile, or just apply the handy patch below.

Patch the source & recompile as usual:

At this point if I was doing a fresh install, I could simply do a make install, but since I already have Xymon installed and working perfectly, I opted to simply copy over the newly patched svcstatus.cgi:

And done!
Thanks to Lubos & Henrik lists.xymon.com/pipermail/xymon/2011-Aug…

Xymon Addons that caught my eye

Posted by & filed under Server Admin, Uncategorized.

I was browsing Xymonton and found a few addons that I may be able to make use of.

IAX2 – www.xymonton.org/monitors:asterisk-iax2
SIP – www.xymonton.org/monitors:asterisk-sip
Mail Graphs – www.bakarasse.de/pages/en/xymon-monitori…
Windows Update/Reboot Test – www.xymonton.org/monitors:winupd
APC UPS Monitoring & Temperature – developer.sysco.ch/hobbit/
HP Proliant HW RAID Monitoring – code.google.com/p/hphwraid/

Xymon – drop a test or host

Posted by & filed under Server Admin.

Straight from the manpage:

Deleting a test:

For Xymon use the command:

/server/bin/xymon 127.0.0.1 “drop HOSTNAME ftp”

For BigBrother use the command

~/server/bin/bb 127.0.0.1 “drop HOSTNAME ftp”

to permanenly remove all traces of a test. Note that you need the quotes around the “drop HOSTNAME ftp”.

 

For Xymon: Deleting a host

First, remove the host from the /etc/xymon/hosts.conf file. Then use the command

server/bin/xymon 127.0.0.1 “drop HOSTNAME”

to permanently remove all traces of a host. Note that you need the quotes around the “drop HOSTNAME”.

 

For Big Brother: Deleting a host

First, remove the host from the ~/server/etc/bb-hosts file. Then use the command

~/server/bin/bb 127.0.0.1 “drop HOSTNAME”

to permanently remove all traces of a host. Note that you need the quotes around the “drop HOSTNAME”.

 

Updated 03/24/14 w/ Xymon specific syntax.

.htaccess and Windows — oh noes!

Posted by & filed under Server Admin.

Windows sucks. But sometimes I have to deal with an even bigger abomination — Linux software on a Windows boxen! Today I had to get a .htaccess file working, and since unix paths can get a bit tricky on windows, I figured I’d jot down the steps:

In the htaccess.conf we need to do the following:

  1. Enable mod_rewrite in the LoadModule section
  2. Set “AllowOverride All”

I was using vhosts in this instance, so I needed to add the following to the appropriate <VirtualHost> section:

So now the entire <VirtualHost> block looks like:

Lastly, my .htaccess file is pretty much setup like normal:

The .htpasswd file can be generated using htpasswd — Apache packages the htpasswd in the bin directory. It will likely be somewhere like: C:\Program Files\Apache Software Foundation\Apache2.2\bin

Toss the resulting file int he path defined by AuthUserFile in the .htaccess file and you are done!