OT: What happens when your Google Drive fills up

Area for non-Galaxy Rangers discussions.
Post Reply
User avatar
RabbiBob
Site Admin
Posts: 236
Joined: Tue Feb 06, 2018 2:07 pm
Location: BETA Mountain
Contact:

OT: What happens when your Google Drive fills up

Post by RabbiBob » Sun Sep 08, 2019 3:50 pm

Might be a good idea to check if a process is running before spawning another process of the same type.

I have a new security camera system that we bought before we took our Scotland trip and one of the things I noticed was that it was hard to determine when\if something occurred. I ended up setting the unit to email movement jpg pictures to an email address and then I have a couple of scripts to check the email address, download the files, sort them, then ship them up to Google Drive. In this fashion I can visually look at a day on a mobile device, then to go to video if needed. I can also assemble time lapse videos from the pictures.

The scripts, which I need to document someday:

Code: Select all

fetchmail
cp /var/spool/mail/SOMEUSER /home/SOMEUSER/
echo 'd *' | mail -N
ripmime -i SOMEUSER -d /home/SOMEUSER/checkmail
rm /home/SOMEUSER/checkmail/text*
perl /home/SOMEUSER/movefiles.pl
rclone copy /home/SOMEUSER/securitycameras rclone:securitycameras --transfers=40 --checkers=15 --tpslimit=10 --drive-chunk-size=1M --bwlimit 4M
  • Check the mail
  • Process the mail

Code: Select all

use strict;
use warnings;
use English;
use File::Basename qw( fileparse );
use File::Path qw( make_path );
use File::Spec;
use File::Copy;

my $dir = '/home/SOMEUSER/checkmail';
my $destination ='/home/SOMEUSER/securitycameras/'; #softlink
foreach my $fp (glob("$dir/*.jpg")) {
#  printf "%s\n", $fp;

print "_______________________________________\n";
print "Source File: $fp\n";

my $filename_year="";
my $filename_month="";
my $filename_day="";
my $filename_camera="";
my $filename_time="";
my $filename_extension="";
my $output_filename="";
#04_20190430185212.jpg

if ( $fp =~ /(\d\d)_(\d{4})(\d{2})(\d{2})(.*)(.jpg)/ ) {
#print "$1\n";
$filename_year=$2;
$filename_month=$3;
$filename_day=$4;
$filename_camera=$1;
$filename_time=$5;
$filename_extension=$6;
$output_filename=$filename_year.$filename_month.$filename_day.'_'.$filename_time.'_'.$filename_camera.$filename_extension;
print "Output File: $output_filename\n";
}
### Check for destination directory - YEAR
my $dest_year = $destination.$filename_year;
print "Dest Year: $dest_year\n";

### Check for destination directory - MONTH
my $dest_month = $dest_year.'/'.$filename_month;
print "Dest Month: $dest_month\n"; 

### Check for destination directory - DAY
my $dest_day = $dest_month.'/'.$filename_day;
print "Dest Day: $dest_day\n";

### Dest File

my $dest_file = $dest_day.'/'.$output_filename;
print "Dest File: $dest_file\n";

############# Create Dirs ################

	if ( !-d $dest_year ) 
		{
		print "Creating $dest_year\n";
    		make_path $dest_year or die "Failed to create path: $dest_year";
		}
	if ( !-d $dest_month ) 
		{
		print "Creating $dest_month\n";	
    		make_path $dest_month or die "Failed to create path: $dest_month";
		}
	if ( !-d $dest_day ) 
		{
		print "Creating $dest_day\n";
    		make_path $dest_day or die "Failed to create path: $dest_day";
		}

	#### MOVE FILES #####
	move ($fp,$dest_file) or die "The move operation failed: $!";
	}

  • Move the attachments with Perl
  • Upload to Google Drive
Seems pretty simple and I run this every 5 minutes and I gave it some, I think, fairly wide authority to do what it needs to do.

Code: Select all

rclone copy /home/SOMEUSER/securitycameras rclone:securitycameras --transfers=40 --checkers=15 --tpslimit=10 --drive-chunk-size=1M --bwlimit 4M
Around 2 AM on Sept 06, the server was non-responsive (once I figured out what it was, I looked back to see when the last picture was downloaded).

At the time, nothing would load. I was on vacation camping, without much connectivity, and well I was on vacation.

The server was silent.

Until midnight Sunday morning: I had a process that runs weekly that failed for the first time in almost two years and it emailed me. The server was still running?

Here is what I currently believe happened:
  • The cameras uploaded its usual cache of files
  • The script fired off and started checking pictures
  • It did not complete the run in the five minutes
  • 5 minutes later, another firing of the script occurred
  • 5 minutes later, another firing of the script occurred
  • 5 minutes later, another firing of the script occurred
  • 5 minutes later, another firing of the script occurred
Arrived home on Sunday and could only ping the server. Cold rebooted the server. It restarted and was good for an hour, then I started to see the slow downs again. I ended up logging in locally and killing a slew of rclone process that were all fighting each other to sync

Code: Select all

sudo kill -9 2474
sudo kill -9 3074
sudo kill -9 1810
sudo kill -9 1890
sudo kill -9 2095
sudo kill -9 2225
sudo kill -9 2321
sudo kill -9 2979
sudo kill -9 3564
sudo kill -9 3670
sudo kill -9 3687
sudo kill -9 3298
sudo kill -9 1663
sudo kill -9 2000
sudo kill -9 3468
sudo kill -9 4282
sudo kill -9 3181
sudo kill -9 1540
sudo kill -9 4520
sudo kill -9 2583
sudo kill -9 2708
The CPU was at 100% (the non-responsive issue).

I killed the 5 minute script and just had one running to watch what happens...

Two things came up:

1) What I need to do is rewrite the script to check to see if rclone is ALREADY running, and if so, not fire another.

Code: Select all

ps -C rclone >/dev/null && echo "Running" || echo "Not running"
Running
2) When you max out your Google Drive account space, fun things happen

Code: Select all

Output from command sh /home/SOMEUSER/processmail.sh ..
fetchmail: another foreground fetchmail is running at 6589.

Held 0 messages in /var/mail/SOMEUSER
rm: cannot remove '/home/SOMEUSER/checkmail/text*': No such file or directory
_______________________________________
Source File: /home/SOMEUSER/checkmail/03_20190905214659.jpg
Output File: 20190905_214659_03.jpg
Dest Year: /home/SOMEUSER/securitycameras/2019
Dest Month: /home/SOMEUSER/securitycameras/2019/09
Dest Day: /home/SOMEUSER/securitycameras/2019/09/05
Dest File: /home/SOMEUSER/securitycameras/2019/09/05/20190905_214659_03.jpg
_______________________________________
Source File: /home/SOMEUSER/checkmail/04_20190905222832.jpg
Output File: 20190905_222832_04.jpg
Dest Year: /home/SOMEUSER/securitycameras/2019
Dest Month: /home/SOMEUSER/securitycameras/2019/09
Dest Day: /home/SOMEUSER/securitycameras/2019/09/05
Dest File: /home/SOMEUSER/securitycameras/2019/09/05/20190905_222832_04.jpg
2019/09/08 15:41:12 NOTICE: 2019/06/02/20190602_150052_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:12 NOTICE: 2019/06/02/20190602_160136_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:12 NOTICE: 2019/06/02/20190602_160136_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:12 NOTICE: 2019/06/02/20190602_163429_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:28 ERROR : 2019/09/04/20190904_091509_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:31 ERROR : 2019/09/04/20190904_090050_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:33 ERROR : 2019/09/04/20190904_091707_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:33 ERROR : 2019/09/04/20190904_092916_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:34 NOTICE: 2019/07/07/20190707_023330_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:36 ERROR : 2019/09/04/20190904_093439_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:37 ERROR : 2019/09/04/20190904_094000_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:38 NOTICE: 2019/05/15/20190515_013646_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:38 NOTICE: 2019/05/15/20190515_024904_04.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:38 NOTICE: 2019/05/15/20190515_042640_01.jpg: Duplicate object found in destination - ignoring
2019/09/08 15:41:39 ERROR : 2019/09/04/20190904_093956_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:39 ERROR : 2019/09/04/20190904_095442_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:42 ERROR : 2019/09/04/20190904_100755_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:43 ERROR : 2019/09/04/20190904_100033_04.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:43 ERROR : 2019/09/04/20190904_101654_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2019/09/08 15:41:46 ERROR : 2019/09/04/20190904_102406_02.jpg: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded

That error is why there were compounded rclones running: it couldn't ever fully sync due to file quota being exceeded and every time it tried it would take a long time to error on each file (thousands, 15GB worth) and a new rclone would spawn. After awhile, all of the server resources would be depleted and it would come to a crawl.

I need to delete some files to free up some space.

*sigh*

Hope you at least enjoyed the cats time lapse!

User avatar
RabbiBob
Site Admin
Posts: 236
Joined: Tue Feb 06, 2018 2:07 pm
Location: BETA Mountain
Contact:

Re: OT: What happens when your Google Drive fills up

Post by RabbiBob » Mon Sep 09, 2019 8:27 pm

Code: Select all

if ps ax | grep -v grep | grep -v grep | grep rclone > /dev/null
then
    echo "RCLONE is running, let's not bind the system up"
else
    echo "RCLONE is not running"
fetchmail
cp /var/spool/mail/SOMEUSER /home/SOMEUSER/
echo 'd *' | mail -N
ripmime -i SOMEUSER -d /home/SOMEUSER/checkmail
rm /home/SOMEUSER/checkmail/text*
perl /home/SOMEUSER/movefiles.pl
# rclone copy /home/SOMEUSER/securitycameras rclone:securitycameras --transfers=40 --checkers=15 --tpslimit=10 --drive-chunk-size=1M --bwlimit 4M
rclone copy /home/SOMEUSER/securitycameras rclone:securitycameras --transfers=4 --checkers=2 --tpslimit=2 --drive-chunk-size=1M --bwlimit 4M
fi
I have about three months to figure out how to remove old pictures in some automated fashion from Google Drive, but for now at least the system won't get bound up.

Code: Select all

rclone sync /home/SOMEUSER/securitycameras rclone:securitycameras --transfers=4 --checkers=2 --tpslimit=2 --drive-chunk-size=1M --bwlimit 4M
Changed copy to sync, that way I can clean up by moving\deleting files locally.

:doc:

User avatar
MaryNiko
Slaverlord
Posts: 171
Joined: Sun Nov 11, 2018 10:08 pm

Re: OT: What happens when your Google Drive fills up

Post by MaryNiko » Tue Sep 10, 2019 9:01 am

I have not understood anything! Lol.
well yes, now it will not be blocked and that you have extra work here.
we could do a reboot of GR with "RabbiDoc" ... :lol: :D
Attachments
Screenshot_2019-06-12-00-26-10.jpg
Screenshot_2019-06-12-00-26-10.jpg (58.28 KiB) Viewed 2371 times

Post Reply