If you’re like me, then you’ve probably set up your Linux server completely wrong! Seriously though, many people who host with Linux probably have some hosting company set their server up. Unless your admins took the extra care to automatically separate your log files by date, then you may have noticed one huge-ass log file in your web page’s directory… especially if it isn’t compressed.
In my case I had an 11.3GB log file sitting in my home directory, and I really needed to split it up into smaller files.
There’s a very easy way to do this with the Linux command line.
Before you can split the file up, you need to go to the path of the log file folder. Depending on how your Linux server is set up, it may be in /var/log/httpd/ or /home/<username>/.
If you actually look at one of the log files, you will see several bits of information. You will notice that the date and time is recorded with each action in this format: [29/Mar/2012:12:00:00 -0700]. Knowing how the date is stored will help you split the files up.
To split by month (in this case March 2012), type this command:
grep -e "Mar/2012" access_log > mar_2012.log
You are basically searching the log file for any entries that have “Mar/2012” in them and saving the output to a new file called “mar_2012.log” in the same directory. If you have a really large file then this could take several minutes.
Now that we know how to split the file by month, it’s rather easy to figure out how to split the file into daily chunks. This would be a very tedious process, but you probably have nothing but time on your hands if you’re actually reading this article.
To split by day, type this command:
grep -e "29/Mar/2012" access_log > 29_mar_2012.log
Of course you can change the output file name to any format you desire. Once you split all of your files you can either compress the original file or delete it. Then you can compress all of your smaller files and download them and archive them away.