[Cialug] Is there a better way to sync two directories?

[Josh More]

Whoops.  Didn't read the whole thing.  Sorry about the rsync suggestion.

I'd do what Barry suggested.  Specifically, I would launch a shell to
each server and run:

find . -type f -exec cksum '{}' \;

on each box.  Then I'd compare the outputs and grep on ">" to identify
all files that are different and pull only from the first list.  Then
I'd pipe that through cut to get the file list:

diff 1 2 | grep '>' | cut -d ' ' -f 4-999 > copy.list

Then I'd feed that list into scp through a loop:

for i in `cat copy.list`; do scp $i $user at host:/; done


Note, this may break if you have files with spaces in the names.  Test
stuff first.  I *think* that the problem would come in at the for loop
stage, so you might want to run tr on copy.list before you parse it
with cat.

There is probably a more elegant solution where you could do it in
fewer steps and it's more tolerant of odd file names.

-Josh






On Mon, Jan 2, 2012 at 4:14 PM, Adam Shannon <adam at ashannon.us> wrote:
> I have a shell on both boxes, however my 1and1 setup won't allow me to
> run executables. What I'm doing is sitting in my amazon box with a
> manual rsync command to pull the 1and1 file list so it can pull down
> the new images.
>
> Thanks Josh for the tip, by the way.
>
> On Mon, Jan 2, 2012 at 16:10, Barry Von Ahsen <barry at vonahsen.com> wrote:
>> do you have shell on either?  could you upload a statically compiled rsync?
>>
>>
>> if not, you could generate a file list on each machine, diff it, and I'm
>> pretty sure scp/sftp can take a file of files to transfer
>>
>>
>> there's also unison, I think that will work from your local machine to
>> sync two remote machines
>>
>> -barry
>>
>>
>>
>> On 01/02/2012 02:20 PM, Adam Shannon wrote:
>>> Hello all,
>>>
>>> I'm moving web hosts (from a restricted 1and1 box to an amazon
>>> instance) and I was wondering if there was a better way to copy all of
>>> the files over. The biggest part is my image hosting site, which I
>>> would like to keep the amazon box as close as I can to my 1and1 box.
>>> However, on my 1and1 box I don't have access to install programs (so
>>> rsync is out). What I'm doing to start is using scp to transfer
>>> everything over, however it looks like scp will copy files even if
>>> they already exist. With 16,000+ images I'd rather not do that (if I
>>> want to keep the two boxes as similar as possible during the move).
>>>
>>> My question is, is there a better way to ensure the two boxes are the
>>> same even on this limited setup?
>>>
>>> Thanks,
>>>
>>
>> _______________________________________________
>> Cialug mailing list
>> Cialug at cialug.org
>> http://cialug.org/mailman/listinfo/cialug
>
>
>
> --
> Adam Shannon
> Web Developer
> University of Northern Iowa
> Sophomore -- Computer Science B.S. & Mathematics
> http://ashannon.us
> _______________________________________________
> Cialug mailing list
> Cialug at cialug.org
> http://cialug.org/mailman/listinfo/cialug