I have a new security camera system that we bought before we took our Scotland trip and one of the things I noticed was that it was hard to determine when\if something occurred. I ended up setting the unit to email movement jpg pictures to an email address and then I have a couple of scripts to check the email address, download the files, sort them, then ship them up to Google Drive. In this fashion I can visually look at a day on a mobile device, then to go to video if needed. I can also assemble time lapse videos from the pictures.
The scripts, which I need to document someday:
Code: Select all
fetchmail
cp /var/spool/mail/SOMEUSER /home/SOMEUSER/
echo 'd *' | mail -N
ripmime -i SOMEUSER -d /home/SOMEUSER/checkmail
rm /home/SOMEUSER/checkmail/text*
perl /home/SOMEUSER/movefiles.pl
rclone copy /home/SOMEUSER/securitycameras rclone:securitycameras --transfers=40 --checkers=15 --tpslimit=10 --drive-chunk-size=1M --bwlimit 4M
- Check the mail
- Process the mail
Code: Select all
use strict;
use warnings;
use English;
use File::Basename qw( fileparse );
use File::Path qw( make_path );
use File::Spec;
use File::Copy;
my $dir = '/home/SOMEUSER/checkmail';
my $destination ='/home/SOMEUSER/securitycameras/'; #softlink
foreach my $fp (glob("$dir/*.jpg")) {
# printf "%s\n", $fp;
print "_______________________________________\n";
print "Source File: $fp\n";
my $filename_year="";
my $filename_month="";
my $filename_day="";
my $filename_camera="";
my $filename_time="";
my $filename_extension="";
my $output_filename="";
#04_20190430185212.jpg
if ( $fp =~ /(\d\d)_(\d{4})(\d{2})(\d{2})(.*)(.jpg)/ ) {
#print "$1\n";
$filename_year=$2;
$filename_month=$3;
$filename_day=$4;
$filename_camera=$1;
$filename_time=$5;
$filename_extension=$6;
$output_filename=$filename_year.$filename_month.$filename_day.'_'.$filename_time.'_'.$filename_camera.$filename_extension;
print "Output File: $output_filename\n";
}
### Check for destination directory - YEAR
my $dest_year = $destination.$filename_year;
print "Dest Year: $dest_year\n";
### Check for destination directory - MONTH
my $dest_month = $dest_year.'/'.$filename_month;
print "Dest Month: $dest_month\n";
### Check for destination directory - DAY
my $dest_day = $dest_month.'/'.$filename_day;
print "Dest Day: $dest_day\n";
### Dest File
my $dest_file = $dest_day.'/'.$output_filename;
print "Dest File: $dest_file\n";
############# Create Dirs ################
if ( !-d $dest_year )
{
print "Creating $dest_year\n";
make_path $dest_year or die "Failed to create path: $dest_year";
}
if ( !-d $dest_month )
{
print "Creating $dest_month\n";
make_path $dest_month or die "Failed to create path: $dest_month";
}
if ( !-d $dest_day )
{
print "Creating $dest_day\n";
make_path $dest_day or die "Failed to create path: $dest_day";
}
#### MOVE FILES #####
move ($fp,$dest_file) or die "The move operation failed: $!";
}
- Move the attachments with Perl
- Upload to Google Drive
Code: Select all
rclone copy /home/SOMEUSER/securitycameras rclone:securitycameras --transfers=40 --checkers=15 --tpslimit=10 --drive-chunk-size=1M --bwlimit 4M
At the time, nothing would load. I was on vacation camping, without much connectivity, and well I was on vacation.
The server was silent.
Until midnight Sunday morning: I had a process that runs weekly that failed for the first time in almost two years and it emailed me. The server was still running?
Here is what I currently believe happened:
- The cameras uploaded its usual cache of files
- The script fired off and started checking pictures
- It did not complete the run in the five minutes
- 5 minutes later, another firing of the script occurred
- 5 minutes later, another firing of the script occurred
- 5 minutes later, another firing of the script occurred
- 5 minutes later, another firing of the script occurred
Code: Select all
sudo kill -9 2474
sudo kill -9 3074
sudo kill -9 1810
sudo kill -9 1890
sudo kill -9 2095
sudo kill -9 2225
sudo kill -9 2321
sudo kill -9 2979
sudo kill -9 3564
sudo kill -9 3670
sudo kill -9 3687
sudo kill -9 3298
sudo kill -9 1663
sudo kill -9 2000
sudo kill -9 3468
sudo kill -9 4282
sudo kill -9 3181
sudo kill -9 1540
sudo kill -9 4520
sudo kill -9 2583
sudo kill -9 2708
I killed the 5 minute script and just had one running to watch what happens...
Two things came up:
1) What I need to do is rewrite the script to check to see if rclone is ALREADY running, and if so, not fire another.
Code: Select all
ps -C rclone >/dev/null && echo "Running" || echo "Not running"
Running
Code: Select all
Output from command sh /home/SOMEUSER/processmail.sh ..
fetchmail: another foreground fetchmail is running at 6589.
Held 0 messages in /var/mail/SOMEUSER
rm: cannot remove '/home/SOMEUSER/checkmail/text*': No such file or directory
_______________________________________
Source File: /home/SOMEUSER/checkmail/03_20190905214659.jpg
Output File: 20190905_214659_03.jpg
Dest Year: /home/SOMEUSER/securitycameras/2019
Dest Month: /home/SOMEUSER/securitycameras/2019/09
Dest Day: /home/SOMEUSER/securitycameras/2019/09/05
Dest File: /home/SOMEUSER/securitycameras/2019/09/05/20190905_214659_03.jpg
_______________________________________
Source File: /home/SOMEUSER/checkmail/04_20190905222832.jpg
Output File: 20190905_222832_04.jpg
Dest Year: /home/SOMEUSER/securitycameras/2019
Dest Month: /home/SOMEUSER/securitycameras/2019/09
Dest Day: /home/SOMEUSER/securitycameras/2019/09/05
Dest File: /home/SOMEUSER/securitycameras/2019/09/05/20190905_222832_04.jpg
2019/09/08 15:41:12 NOTICE: 2019/06/02/20190602_150052_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:12 NOTICE: 2019/06/02/20190602_160136_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:12 NOTICE: 2019/06/02/20190602_160136_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:12 NOTICE: 2019/06/02/20190602_163429_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:28 ERROR : 2019/09/04/20190904_091509_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:31 ERROR : 2019/09/04/20190904_090050_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:33 ERROR : 2019/09/04/20190904_091707_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:33 ERROR : 2019/09/04/20190904_092916_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:34 NOTICE: 2019/07/07/20190707_023330_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:36 ERROR : 2019/09/04/20190904_093439_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:37 ERROR : 2019/09/04/20190904_094000_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:38 NOTICE: 2019/05/15/20190515_013646_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:38 NOTICE: 2019/05/15/20190515_024904_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:38 NOTICE: 2019/05/15/20190515_042640_01.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:39 ERROR : 2019/09/04/20190904_093956_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:39 ERROR : 2019/09/04/20190904_095442_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:42 ERROR : 2019/09/04/20190904_100755_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:43 ERROR : 2019/09/04/20190904_100033_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:43 ERROR : 2019/09/04/20190904_101654_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:46 ERROR : 2019/09/04/20190904_102406_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
That error is why there were compounded rclones running: it couldn't ever fully sync due to file quota being exceeded and every time it tried it would take a long time to error on each file (thousands, 15GB worth) and a new rclone would spawn. After awhile, all of the server resources would be depleted and it would come to a crawl.
I need to delete some files to free up some space.
*sigh*
Hope you at least enjoyed the cats time lapse!