[Cialug] Efficiently removing the beginning of a file

Daniel A. Ramaley daniel.ramaley at DRAKE.EDU
Mon May 21 11:04:18 CDT 2007


I have a ~70 MB file. The first 3635 bytes need to be removed. What is 
the most efficient way to do that? I did this, knowing it would work 
but would be slow:
    $ dd if=inputfile of=outputfile ibs=1 obs=1M skip=3635
It did indeed work. But it took 274 seconds (and pegged the CPU the 
entire time), whereas simply copying the file with cp only takes 2 
seconds. Since what i want to do is not *that* different an action from 
just copying the file (at least in terms of the minimum disk operations 
that would be required), it seems to me that there should be a way to 
do it that only takes ~2 seconds. What are some other command line ways 
to do this that would be more efficient?

Actually, before hitting send i tried another test, just flipping the 
"ibs" and "skip" values:
    $ dd if=inputfile of=outputfile2 ibs=3635 obs=1M skip=1
That only took 2.5 seconds, which is much closer to the theoretical 2 
second time that should be possible. But i guess what i'm curious about 
is the general problem; if there is a large file and you need to remove 
some small number of bytes from the beginning of it, how is that best 
accomplished? If i had needed to remove only 1 byte for example, i 
would have had to have used "ibs=1 skip=1" which would have taken 
around 274 seconds again.

------------------------------------------------------------------------
Dan Ramaley                            Dial Center 118, Drake University
Network Programmer/Analyst             2407 Carpenter Ave
+1 515 271-4540                        Des Moines IA 50311 USA


More information about the Cialug mailing list