To Linux Gurus-Limit of # of files in folder-Ext3

Discussion in 'General' started by torontob, Mar 19, 2009.

  1. torontob

    Joined:
    May 18, 2008
    Messages:
    219
    Likes Received:
    0
    Hi Guys,

    So Elastix provides the Monitoring Module and FreePBX allows for Recording at extension level. Has anyone ever faced the problem or thought about what happens to all the files that get recorded into one single folder? I think there is no mechanism to sort files into different folders (correct me if I am wrong). What if the number of recorded calls go over 20, 000 or 30, 000 in /var/spool/asterisk/monitor? Doesn't that introduce a problem? Doesn't file system Ext3 have a limit to the number of files in a folder? Wouldn't this mess up the whole structure and overload the system?

    I use FreePBX Recordings module to sign into each extension and check Voicemail and recorded calls after the Elastix module failed me but even that warns me if I have more than 3000 files in a folder. I changed the limit in php file from 3,000 to 300,000 and now everything works fine in FreePBX ARI Recordings Module but I don't know what is waiting for me in future.

    If I was to sort all these files into differnt folders manually then that means that I lose accessibility and that is really not desireable. I am wondering if anyone else with a call center has faced the same problem and came up with a solution.

    Any feedback on this is much appreciated. Specially from Moderators.

    Thanks,
     
  2. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    Under General settings>Call Recording
    there is a box where you can run a script after each recording is finished, you can do anything you want with it there. like a directory by day for example.
     
  3. torontob

    Joined:
    May 18, 2008
    Messages:
    219
    Likes Received:
    0
    Thanks for the input Dicko.

    Wouldn't doing a script or a cronjob mess up the whole Monitoring section or the FreePBX Recording module? I mean I still want to be able to pull up files normally through the Web GUI interface.

    Thanks,
     
  4. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    Yes it would, but if I remember rightly there is a 32k limit on the inodes of an ext3 file system, even with the patience of Job I don't think you could live with a php interface into that pile of 32k files, one option would be to go to an ext4 fs, others would be archiving older recordings into a more appropriate place and point a more appropriate interface into that structure.
     
  5. torontob

    Joined:
    May 18, 2008
    Messages:
    219
    Likes Received:
    0
    Let's explore both options that you purpose :)

    1- I never did a change from ext3 to ext4. Is that a breeze? or would it require breaking the whole production system and re-installing?

    2- What would you consider a more appropriate interface that I can point to the recordings? I would love to see something simple other than the monitoring module from elastix or freepbx that could simply go into folder and read files.

    And if I were to actually run a script and sort all files by day, or by extension, would there be anyway to get FreePBX recording to read it all from differnt directories? or the Monitoring module to read it from different directories? If "Not" again do you know of any even in-house GUI module that would read files?

    I am assuming I have to go through MySql too to change all the locations?!

    Thanks
     
  6. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    You indicated before a limit of 300000 files and that just won't fly in an ext3 fs.

    EXT4 is not even possible in the current kernel
    (ref:
    http://www.centos.org/modules/newbb/vie ... c_id=18159
    )

    but probably necessary if you want to manage 300000 files, you are way out of FreePBX zone of comfort and probably need a custom designed app. to suit what you are trying to (presumably a call center)
     
  7. torontob

    Joined:
    May 18, 2008
    Messages:
    219
    Likes Received:
    0
    Thanks for the input again. I beleive you have previously adviced me to look into the Call Center modules. I haven't been able to explore it much for two reasons:

    1- It's a beast in itself and requires agent training; something that I can't afford to do currently.
    2- There is no guarrantee that recording would be done any better than it is with FreePBX and Asterisk MixMonitor currently.

    So, maybe I can turn on direcotory indexing or soemthing so that Apache allows for files to display on a page for display. Just hate to get around MySql to pull the data and really I am not familiar with it. Maybe I should post this as a bounty?!
     
  8. torontob

    Joined:
    May 18, 2008
    Messages:
    219
    Likes Received:
    0
    Good Morning Dicko,

    How does the Under General settings>Call Recording>Run a Script work? I mean let's say you write a script to move a file to a folder. How do you harvent the .wav file name? I know that it is Date-Time-UniqueID.wav but if the script is not run from Asterisk CLI, how do you know what the file name was to be moved to a folder?

    Or maybe move all .wav to a folder that is specific for the day? Hence the main folder shouldn't have any .wav file?

    Any suggestions?

    Thanks
     
  9. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
  10. torontob

    Joined:
    May 18, 2008
    Messages:
    219
    Likes Received:
    0
    Thanks. I see that now.

    Forget me for asking such naive questions but I guess I have to tackle this before I overload the system with the many thousands of files. So, this script would run from inside Asterisk CLI and I have to give commands like (in the shell script file called script):

    !touch success

    Or does this run from the shell:

    touch success

    I tried both ways and this is what happens:

    -- Executing [h@macro-dial:1] Macro("SIP/602-0857c070", "hangupcall") in new stack
    -- Executing [s@macro-hangupcall:1] ResetCDR("SIP/602-0857c070", "vw") in new stack
    -- Executing [s@macro-hangupcall:2] NoCDR("SIP/602-0857c070", "") in new stack
    -- Executing [s@macro-hangupcall:3] GotoIf("SIP/602-0857c070", "1?skiprg") in new stack
    -- Goto (macro-hangupcall,s,6)
    -- Executing [s@macro-hangupcall:6] GotoIf("SIP/602-0857c070", "1?skipblkvm") in new stack
    -- Goto (macro-hangupcall,s,9)
    -- Executing [s@macro-hangupcall:9] GotoIf("SIP/602-0857c070", "1?theend") in new stack
    -- Goto (macro-hangupcall,s,11)
    -- Executing [s@macro-hangupcall:11] Hangup("SIP/602-0857c070", "") in new stack
    == Spawn extension (macro-hangupcall, s, 11) exited non-zero on 'SIP/602-0857c070' in macro 'hangupcall'
    == Spawn extension (macro-hangupcall, s, 11) exited non-zero on 'SIP/602-0857c070'
    == End MixMonitor Recording SIP/602-0857c070
    == Executing [/var/spool/asterisk/monitor/test/script]

    And then after the last line it's just stuck and I see no effects coming out of the file script.

    Please enlighten me as to what I am doing wrong.

    Thanks
     
  11. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    Either way

    from bash:

    /var/spool/asterisk/monitor/test/script /var/spool/asterisk/monitor/OUT1234-20090323-223159-1237872718.39.wav

    or indeed from the asterisk CLI
    !/var/spool/asterisk/monitor/test/script /var/spool/asterisk/monitor/OUT1234-20090323-223159-1237872718.39.wav

    where of course /var/spool/asterisk/monitor/OUT1234-20090323-223159-1237872718.39.wav is a REAL file, the script, (whichever language it is written in) will see that as PARAMETER1 ($1 in bash)


    I suggest you get it running from bash with lots of "debug output to terminal" lines inserted first.

    (Both of course one liners if the formatting has split it)
     
  12. torontob

    Joined:
    May 18, 2008
    Messages:
    219
    Likes Received:
    0
    Well see, I am confused as to where this script is run from when I put it under General Tab. My script file only contains the following line just for test (no .wav file movement yet; I trying to prove the concept for now):

    touch success.txt

    Once recording is done I see this in CLI:

    == End MixMonitor Recording SIP/602-0857c070
    == Executing [/var/spool/asterisk/monitor/test/script]

    And that's it. It's stuck on the last line for ever. When I go to see if a success.txt was created, I don't see it meaning the script file didn't run. I also tried the following variation with no luck:

    !touch success.txt

    The file "script" has been given "chmod +x" already and ./script yields the file success.txt in the directory but it just doesn't do the job when run from FreePBX.
     
  13. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    The script (as is all asterisk in Elastix) is run by the user asterisk
    The asterisk user's home directory is /var/lib/asterisk
    check there if you have un-anchored paths in your scripts

    you need to anchor your paths and make sure they are writable by asterisk (the user)

    if /var/spool/asterisk/monitor/test/script (the script that is called by the recording context) exists I suggest you replace touch success.txt with
    touch /var/spool/asterisk/monitor/success.txt as a first step
     
  14. torontob

    Joined:
    May 18, 2008
    Messages:
    219
    Likes Received:
    0
    Hmmm....I can't seem to work it. I still get the same result despite putting the script file in /var/lib/asterisk/ folder or defining full paths. Maybe it's the Asterisk user. How can I give permissions and where should I give permissions? Also, in General Tab I put full path for the script and I also put just script in the field and all of them just go:

    Executing (/var/lib/asterisk/script)
    or
    Executing (/script)

    and that's it.
     
  15. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    Torontob:

    I'm sorry you're having difficulties, but it does work as documented, perhaps a little time spent with a good linux tutorial and a full read of "cmd monitor" at voip-info.org and the digium lists will be productive.

    Baby steps, start with
    echo "torontob wants ^{MIXMON_DIR}^{CALLFILENAME}.^{MIXMON_FORMAT} that started in ^{CHANNEL} to go somehwere else"
    as the command to run, just to convince yourself it is working

    weatch the asterisk CLI output, be aware that there is no terminal assigned to output anything from the script, so all debugging must be done in bash. (by default the shell that the asterisk user has)
    When you have the "parameters" right you can copy the execute line from the CLI and run it in bash (as user asterisk) replacing echo with your script of course. until you succeed ( as long as the file remains where it is of course).
     
  16. torontob

    Joined:
    May 18, 2008
    Messages:
    219
    Likes Received:
    0
    Thanks for all the input Dicko. As you thought it was an Asterisk privilage issue.

    The following did the charm:

    chown -R asterisk.asterisk scriptdirectory
    chown -R asterisk.asterisk script
     
  17. torontob

    Joined:
    May 18, 2008
    Messages:
    219
    Likes Received:
    0
    Now that it seems the script is running, I have the following questions hoping to waste less time on these using your expertise:

    1- What is the easiest way to have the script generate a directory which is in format YearMonthDay. For example, 20090326. The script has to be wise to realise this format in order to be able to perform something like:

    cp /var/spool/asterisk/monitor/^{CALLFILENAME} /var/spool/asterisk/monitor/${YearMonthDay}/

    I tried to explore the CALLFILENAME variable to see if I can extract a YearMonthDay variable but it seems not possible. FreePBX uses the php file:

    /var/lib/asterisk/agi-bin/recordingcheck

    to generate the file name which is in format:

    Inbound Calls (rcin):
    $agi->verbose("CALLFILENAME={$timestamp}-{$uniqueid}");
    $agi->set_variable("CALLFILENAME","{$timestamp}-{$uniqueid}");


    Outbound Calls (rcout):
    $agi->verbose("CALLFILENAME=OUT{$exten['data']}-{$timestamp}-{$uniqueid}");
    $agi->set_variable("CALLFILENAME","OUT{$exten['data']}-{$timestamp}-{$uniqueid}");


    Out exmaple is:
    -- Executing [s@macro-record-enable:999] MixMonitor("SIP/602-0861a770", "20090326-112647-1238081207.4656.wav||/var/spool/asterisk/monitor/test/orgscript") in new stack

    I also tried to use hwclock and clock in linux but the format can't be segregated into YearMonthDay.

    2- As for the variables go, I tried the following in the script and all didn't out the Call File Name to my txt file:

    echo "Call file name:" ^{CALLFILENAME} > txt
    echo "Call file name:" ${CALLFILENAME} > txt
    echo "Call file name: ^{CALLFILENAME}" > txt


    What could be wrong in this case?

    Thanks
    P.S. Hope I haven't given you a headache by now :)
     
  18. ramoncio

    Joined:
    May 12, 2010
    Messages:
    1,663
    Likes Received:
    0
    filename-`date +%Y.%m.%d.%Hh.%mm`.wav
     
  19. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    The callfile name:
    20090326-112647-1238081207.4656.wav
    has the date embedded in itself

    echo $CALLFILENAME|cut -d '-' -f 1
    will extract it

    To be safe you should probably try to "mkdir -p" to each time before writing to it, that will suppress errors if the directory exists
     
  20. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    ??
    echo "Call file name:" ^{CALLFILENAME} > txt
    echo "Call file name:" ${CALLFILENAME} > txt
    echo "Call file name: ^{CALLFILENAME}" > txt

    the parameters used in calling the script will be accessible as $1 $2 etc. in bash.


    #!/bin/bash
    OUTDIR=`echo $1|cut -d '-' -f1`
    mkdir -p /var/spool/asterisk/monitor/$OUTDIR/
    cp(mv?) /var/spool/asterisk/monitor/$1 /var/spool/asterisk/monitor/$OUTDIR/
     

Share This Page