Loading Profile...
Space full???
$ sort -m output500000/* > unity.txt
sort: write failed: /tmp/sortqDVsRW: No space left on device
The files are in /extra/ugrad_space/. I asked it to do that and that was my error.
What am I doing wrong?? Or is it really out of space?
sort: write failed: /tmp/sortqDVsRW: No space left on device
The files are in /extra/ugrad_space/. I asked it to do that and that was my error.
What am I doing wrong?? Or is it really out of space?
I’m wtf
1
person has this question
I have this question, too!
Tell me when someone answers.
The more people who ask this question, the more it gets noticed.
The more people who ask this question, the more it gets noticed.
Create a customer community for your own organization
Plans starting at $19/month
-
Inappropriate?Ah I logged onto a simpsons.ics.uci.edu server and it merged it correctly.
HINTS: I had to delete the _Logs folder and I had to execute the sort from one folder up from the folders my files were stored in.
So my output folder would be like:
>/extra/ugrad_space/leecf/hadoop/output
I delete the _Logs directory, then i type: "up" into console.
>/extra/ugrad_space/leecf/hadoop/output
>up
>/extra/ugrad_space/leecf/hadoop/
>sort -m output/* > unity.txt
I’m merged
-
Inappropriate?You can also copy them onto your local machine with scp and merge them there...
-
Inappropriate?Well I don't have unix so I don't know what the windows equivalent of this command would be.
BTW, How do we "tail" this file for only the last 1000 entries? I don't want to download the whole 3GB, buy an extra 4GB of ram, open it and wait for 3 hours for it to load, and find the last few lines...
-
Inappropriate?You can use a program called "WinSCP"... or cygwin which is basically a unix emulator for windows.
Also, in unix just type, "tail -n 1000 filename > newFilename" where filename is the whole posting list, and newFilename is the name of the file where you want to store the last 1000 lines to
-
Inappropriate?Well I mean, how do I merge these files once I have them on my disk? I have 26 separate files and I need to merge them into one, correct?
Oh, and thanks for the tail. That will save me a lot of time.
EDIT:
mitch I saw that. ;)
-
Inappropriate?Oh, got it. Umm, not with winscp. But cygwin will work. Althought it might not be worth installing it and all that jazz...
-
Inappropriate?Well ya gotta do what ya gotta do I guess... 2 hours until that sucker gets fully downloaded to my hard drive.
Thanks for the help. -
Inappropriate?You can do this:
sort -T /extra/ugrad_space/USERNAME/ -m output500000/* > unity.txt
(Although, it will probably be very slow)
If you connect to the theory.ics.uci.edu machines, they have a folder in
/var/preserve/ and you can use that as a temp folder. -
Inappropriate?Yeah. Lemme know if you use cygwin and need help. Also, you can try compressing it and sending it to yourself using.
to create a compressed file type: tar -cz list_of_files > newFileName.tar.gz
to uncompress it type: tar -xzf newFileName.tar.gz -
Inappropriate?OH GOD DANG IT IM SO STUPID! Why didn't I thnk to compress it before? lol. Thanks for the tip.
-
Inappropriate?One more thing guys: In order to tail properly, I needed this:
>module load coreutils
>tail -n 1000 unity.txt > unity1k.txt
-
Inappropriate?i still get the same error
achislom@hayes05 % cd /extra/ugrad_space/achislom/assignment06
achislom@hayes05 % cd hadoop-0.19.0
achislom@hayes05 % sort -m output/* > myPostingList.txt
sort: write failed: /tmp/sortIruZ9c: No space left on device
achislom@hayes05 % -
Inappropriate?try riviera.uci.edu, that was the machine that did it for me.
-
Inappropriate?on the theory cluster do:
sort -T /var/preserve -m output/* > myPostingList.txt
on FamilyGuy do:
sort -T /scratch -m output/* > myPostingList.txt -
Inappropriate?The problem is that "sort" needs a temporary file to keep your data while it is merging everything. Because the source files are so big, the /tmp drive isn't big enough to hold the temporary files.
Mitch's "-T" flag tells sort to put the temporary files somewhere else. On some machines there is a drive which is big enough to handle the temporary files and on others there is not.
To see how much space is on a drive, try
df -h
that stands for "disk-free" and "-h" asks the program to convert the numbers into things that are easy for "h"umans to read like "42G" rather than 43956832.
EMPLOYEE
