OT: What happens when your Google Drive fills up
Posted: Sun Sep 08, 2019 3:50 pm
Might be a good idea to check if a process is running before spawning another process of the same type.
I have a new security camera system that we bought before we took our Scotland trip and one of the things I noticed was that it was hard to determine when\if something occurred. I ended up setting the unit to email movement jpg pictures to an email address and then I have a couple of scripts to check the email address, download the files, sort them, then ship them up to Google Drive. In this fashion I can visually look at a day on a mobile device, then to go to video if needed. I can also assemble time lapse videos from the pictures.
The scripts, which I need to document someday:
Around 2 AM on Sept 06, the server was non-responsive (once I figured out what it was, I looked back to see when the last picture was downloaded).
At the time, nothing would load. I was on vacation camping, without much connectivity, and well I was on vacation.
The server was silent.
Until midnight Sunday morning: I had a process that runs weekly that failed for the first time in almost two years and it emailed me. The server was still running?
Here is what I currently believe happened:
The CPU was at 100% (the non-responsive issue).
I killed the 5 minute script and just had one running to watch what happens...
Two things came up:
1) What I need to do is rewrite the script to check to see if rclone is ALREADY running, and if so, not fire another.
2) When you max out your Google Drive account space, fun things happen
Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
That error is why there were compounded rclones running: it couldn't ever fully sync due to file quota being exceeded and every time it tried it would take a long time to error on each file (thousands, 15GB worth) and a new rclone would spawn. After awhile, all of the server resources would be depleted and it would come to a crawl.
I need to delete some files to free up some space.
*sigh*
Hope you at least enjoyed the cats time lapse!
I have a new security camera system that we bought before we took our Scotland trip and one of the things I noticed was that it was hard to determine when\if something occurred. I ended up setting the unit to email movement jpg pictures to an email address and then I have a couple of scripts to check the email address, download the files, sort them, then ship them up to Google Drive. In this fashion I can visually look at a day on a mobile device, then to go to video if needed. I can also assemble time lapse videos from the pictures.
The scripts, which I need to document someday:
Code: Select all
fetchmail
cp /var/spool/mail/SOMEUSER /home/SOMEUSER/
echo 'd *' | mail -N
ripmime -i SOMEUSER -d /home/SOMEUSER/checkmail
rm /home/SOMEUSER/checkmail/text*
perl /home/SOMEUSER/movefiles.pl
rclone copy /home/SOMEUSER/securitycameras rclone:securitycameras --transfers=40 --checkers=15 --tpslimit=10 --drive-chunk-size=1M --bwlimit 4M
- Check the mail
- Process the mail
Code: Select all
use strict;
use warnings;
use English;
use File::Basename qw( fileparse );
use File::Path qw( make_path );
use File::Spec;
use File::Copy;
my $dir = '/home/SOMEUSER/checkmail';
my $destination ='/home/SOMEUSER/securitycameras/'; #softlink
foreach my $fp (glob("$dir/*.jpg")) {
# printf "%s\n", $fp;
print "_______________________________________\n";
print "Source File: $fp\n";
my $filename_year="";
my $filename_month="";
my $filename_day="";
my $filename_camera="";
my $filename_time="";
my $filename_extension="";
my $output_filename="";
#04_20190430185212.jpg
if ( $fp =~ /(\d\d)_(\d{4})(\d{2})(\d{2})(.*)(.jpg)/ ) {
#print "$1\n";
$filename_year=$2;
$filename_month=$3;
$filename_day=$4;
$filename_camera=$1;
$filename_time=$5;
$filename_extension=$6;
$output_filename=$filename_year.$filename_month.$filename_day.'_'.$filename_time.'_'.$filename_camera.$filename_extension;
print "Output File: $output_filename\n";
}
### Check for destination directory - YEAR
my $dest_year = $destination.$filename_year;
print "Dest Year: $dest_year\n";
### Check for destination directory - MONTH
my $dest_month = $dest_year.'/'.$filename_month;
print "Dest Month: $dest_month\n";
### Check for destination directory - DAY
my $dest_day = $dest_month.'/'.$filename_day;
print "Dest Day: $dest_day\n";
### Dest File
my $dest_file = $dest_day.'/'.$output_filename;
print "Dest File: $dest_file\n";
############# Create Dirs ################
if ( !-d $dest_year )
{
print "Creating $dest_year\n";
make_path $dest_year or die "Failed to create path: $dest_year";
}
if ( !-d $dest_month )
{
print "Creating $dest_month\n";
make_path $dest_month or die "Failed to create path: $dest_month";
}
if ( !-d $dest_day )
{
print "Creating $dest_day\n";
make_path $dest_day or die "Failed to create path: $dest_day";
}
#### MOVE FILES #####
move ($fp,$dest_file) or die "The move operation failed: $!";
}
- Move the attachments with Perl
- Upload to Google Drive
Code: Select all
rclone copy /home/SOMEUSER/securitycameras rclone:securitycameras --transfers=40 --checkers=15 --tpslimit=10 --drive-chunk-size=1M --bwlimit 4M
At the time, nothing would load. I was on vacation camping, without much connectivity, and well I was on vacation.
The server was silent.
Until midnight Sunday morning: I had a process that runs weekly that failed for the first time in almost two years and it emailed me. The server was still running?
Here is what I currently believe happened:
- The cameras uploaded its usual cache of files
- The script fired off and started checking pictures
- It did not complete the run in the five minutes
- 5 minutes later, another firing of the script occurred
- 5 minutes later, another firing of the script occurred
- 5 minutes later, another firing of the script occurred
- 5 minutes later, another firing of the script occurred
Code: Select all
sudo kill -9 2474
sudo kill -9 3074
sudo kill -9 1810
sudo kill -9 1890
sudo kill -9 2095
sudo kill -9 2225
sudo kill -9 2321
sudo kill -9 2979
sudo kill -9 3564
sudo kill -9 3670
sudo kill -9 3687
sudo kill -9 3298
sudo kill -9 1663
sudo kill -9 2000
sudo kill -9 3468
sudo kill -9 4282
sudo kill -9 3181
sudo kill -9 1540
sudo kill -9 4520
sudo kill -9 2583
sudo kill -9 2708
I killed the 5 minute script and just had one running to watch what happens...
Two things came up:
1) What I need to do is rewrite the script to check to see if rclone is ALREADY running, and if so, not fire another.
Code: Select all
ps -C rclone >/dev/null && echo "Running" || echo "Not running"
Running
Code: Select all
Output from command sh /home/SOMEUSER/processmail.sh ..
fetchmail: another foreground fetchmail is running at 6589.
Held 0 messages in /var/mail/SOMEUSER
rm: cannot remove '/home/SOMEUSER/checkmail/text*': No such file or directory
_______________________________________
Source File: /home/SOMEUSER/checkmail/03_20190905214659.jpg
Output File: 20190905_214659_03.jpg
Dest Year: /home/SOMEUSER/securitycameras/2019
Dest Month: /home/SOMEUSER/securitycameras/2019/09
Dest Day: /home/SOMEUSER/securitycameras/2019/09/05
Dest File: /home/SOMEUSER/securitycameras/2019/09/05/20190905_214659_03.jpg
_______________________________________
Source File: /home/SOMEUSER/checkmail/04_20190905222832.jpg
Output File: 20190905_222832_04.jpg
Dest Year: /home/SOMEUSER/securitycameras/2019
Dest Month: /home/SOMEUSER/securitycameras/2019/09
Dest Day: /home/SOMEUSER/securitycameras/2019/09/05
Dest File: /home/SOMEUSER/securitycameras/2019/09/05/20190905_222832_04.jpg
2019/09/08 15:41:12 NOTICE: 2019/06/02/20190602_150052_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:12 NOTICE: 2019/06/02/20190602_160136_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:12 NOTICE: 2019/06/02/20190602_160136_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:12 NOTICE: 2019/06/02/20190602_163429_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:28 ERROR : 2019/09/04/20190904_091509_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:31 ERROR : 2019/09/04/20190904_090050_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:33 ERROR : 2019/09/04/20190904_091707_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:33 ERROR : 2019/09/04/20190904_092916_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:34 NOTICE: 2019/07/07/20190707_023330_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:36 ERROR : 2019/09/04/20190904_093439_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:37 ERROR : 2019/09/04/20190904_094000_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:38 NOTICE: 2019/05/15/20190515_013646_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:38 NOTICE: 2019/05/15/20190515_024904_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:38 NOTICE: 2019/05/15/20190515_042640_01.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:39 ERROR : 2019/09/04/20190904_093956_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:39 ERROR : 2019/09/04/20190904_095442_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:42 ERROR : 2019/09/04/20190904_100755_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:43 ERROR : 2019/09/04/20190904_100033_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:43 ERROR : 2019/09/04/20190904_101654_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:46 ERROR : 2019/09/04/20190904_102406_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
That error is why there were compounded rclones running: it couldn't ever fully sync due to file quota being exceeded and every time it tried it would take a long time to error on each file (thousands, 15GB worth) and a new rclone would spawn. After awhile, all of the server resources would be depleted and it would come to a crawl.
I need to delete some files to free up some space.
*sigh*
Hope you at least enjoyed the cats time lapse!