To Linux Gurus-Limit of # of files in folder-Ext3

torontob

Joined
May 18, 2008
Messages
219
Likes
0
Points
0
#1
Hi Guys,

So Elastix provides the Monitoring Module and FreePBX allows for Recording at extension level. Has anyone ever faced the problem or thought about what happens to all the files that get recorded into one single folder? I think there is no mechanism to sort files into different folders (correct me if I am wrong). What if the number of recorded calls go over 20, 000 or 30, 000 in /var/spool/asterisk/monitor? Doesn't that introduce a problem? Doesn't file system Ext3 have a limit to the number of files in a folder? Wouldn't this mess up the whole structure and overload the system?

I use FreePBX Recordings module to sign into each extension and check Voicemail and recorded calls after the Elastix module failed me but even that warns me if I have more than 3000 files in a folder. I changed the limit in php file from 3,000 to 300,000 and now everything works fine in FreePBX ARI Recordings Module but I don't know what is waiting for me in future.

If I was to sort all these files into differnt folders manually then that means that I lose accessibility and that is really not desireable. I am wondering if anyone else with a call center has faced the same problem and came up with a solution.

Any feedback on this is much appreciated. Specially from Moderators.

Thanks,
 

dicko

Joined
Oct 24, 2008
Messages
4,099
Likes
0
Points
0
#2
Under General settings>Call Recording
there is a box where you can run a script after each recording is finished, you can do anything you want with it there. like a directory by day for example.
 

torontob

Joined
May 18, 2008
Messages
219
Likes
0
Points
0
#3
Thanks for the input Dicko.

Wouldn't doing a script or a cronjob mess up the whole Monitoring section or the FreePBX Recording module? I mean I still want to be able to pull up files normally through the Web GUI interface.

Thanks,
 

dicko

Joined
Oct 24, 2008
Messages
4,099
Likes
0
Points
0
#4
Yes it would, but if I remember rightly there is a 32k limit on the inodes of an ext3 file system, even with the patience of Job I don't think you could live with a php interface into that pile of 32k files, one option would be to go to an ext4 fs, others would be archiving older recordings into a more appropriate place and point a more appropriate interface into that structure.
 

torontob

Joined
May 18, 2008
Messages
219
Likes
0
Points
0
#5
Let's explore both options that you purpose :)

1- I never did a change from ext3 to ext4. Is that a breeze? or would it require breaking the whole production system and re-installing?

2- What would you consider a more appropriate interface that I can point to the recordings? I would love to see something simple other than the monitoring module from elastix or freepbx that could simply go into folder and read files.

And if I were to actually run a script and sort all files by day, or by extension, would there be anyway to get FreePBX recording to read it all from differnt directories? or the Monitoring module to read it from different directories? If "Not" again do you know of any even in-house GUI module that would read files?

I am assuming I have to go through MySql too to change all the locations?!

Thanks
 

dicko

Joined
Oct 24, 2008
Messages
4,099
Likes
0
Points
0
#6
You indicated before a limit of 300000 files and that just won't fly in an ext3 fs.

EXT4 is not even possible in the current kernel
(ref:
http://www.centos.org/modules/newbb/vie ... c_id=18159
)

but probably necessary if you want to manage 300000 files, you are way out of FreePBX zone of comfort and probably need a custom designed app. to suit what you are trying to (presumably a call center)
 

torontob

Joined
May 18, 2008
Messages
219
Likes
0
Points
0
#7
Thanks for the input again. I beleive you have previously adviced me to look into the Call Center modules. I haven't been able to explore it much for two reasons:

1- It's a beast in itself and requires agent training; something that I can't afford to do currently.
2- There is no guarrantee that recording would be done any better than it is with FreePBX and Asterisk MixMonitor currently.

So, maybe I can turn on direcotory indexing or soemthing so that Apache allows for files to display on a page for display. Just hate to get around MySql to pull the data and really I am not familiar with it. Maybe I should post this as a bounty?!
 

torontob

Joined
May 18, 2008
Messages
219
Likes
0
Points
0
#8
Good Morning Dicko,

How does the Under General settings>Call Recording>Run a Script work? I mean let's say you write a script to move a file to a folder. How do you harvent the .wav file name? I know that it is Date-Time-UniqueID.wav but if the script is not run from Asterisk CLI, how do you know what the file name was to be moved to a folder?

Or maybe move all .wav to a folder that is specific for the day? Hence the main folder shouldn't have any .wav file?

Any suggestions?

Thanks
 

dicko

Joined
Oct 24, 2008
Messages
4,099
Likes
0
Points
0
#9

torontob

Joined
May 18, 2008
Messages
219
Likes
0
Points
0
#10
Thanks. I see that now.

Forget me for asking such naive questions but I guess I have to tackle this before I overload the system with the many thousands of files. So, this script would run from inside Asterisk CLI and I have to give commands like (in the shell script file called script):

!touch success

Or does this run from the shell:

touch success

I tried both ways and this is what happens:

-- Executing [h@macro-dial:1] Macro("SIP/602-0857c070", "hangupcall") in new stack
-- Executing [s@macro-hangupcall:1] ResetCDR("SIP/602-0857c070", "vw") in new stack
-- Executing [s@macro-hangupcall:2] NoCDR("SIP/602-0857c070", "") in new stack
-- Executing [s@macro-hangupcall:3] GotoIf("SIP/602-0857c070", "1?skiprg") in new stack
-- Goto (macro-hangupcall,s,6)
-- Executing [s@macro-hangupcall:6] GotoIf("SIP/602-0857c070", "1?skipblkvm") in new stack
-- Goto (macro-hangupcall,s,9)
-- Executing [s@macro-hangupcall:9] GotoIf("SIP/602-0857c070", "1?theend") in new stack
-- Goto (macro-hangupcall,s,11)
-- Executing [s@macro-hangupcall:11] Hangup("SIP/602-0857c070", "") in new stack
== Spawn extension (macro-hangupcall, s, 11) exited non-zero on 'SIP/602-0857c070' in macro 'hangupcall'
== Spawn extension (macro-hangupcall, s, 11) exited non-zero on 'SIP/602-0857c070'
== End MixMonitor Recording SIP/602-0857c070
== Executing [/var/spool/asterisk/monitor/test/script]

And then after the last line it's just stuck and I see no effects coming out of the file script.

Please enlighten me as to what I am doing wrong.

Thanks
 

dicko

Joined
Oct 24, 2008
Messages
4,099
Likes
0
Points
0
#11
Either way

from bash:

/var/spool/asterisk/monitor/test/script /var/spool/asterisk/monitor/OUT1234-20090323-223159-1237872718.39.wav

or indeed from the asterisk CLI
!/var/spool/asterisk/monitor/test/script /var/spool/asterisk/monitor/OUT1234-20090323-223159-1237872718.39.wav

where of course /var/spool/asterisk/monitor/OUT1234-20090323-223159-1237872718.39.wav is a REAL file, the script, (whichever language it is written in) will see that as PARAMETER1 ($1 in bash)


I suggest you get it running from bash with lots of "debug output to terminal" lines inserted first.

(Both of course one liners if the formatting has split it)
 

torontob

Joined
May 18, 2008
Messages
219
Likes
0
Points
0
#12
Well see, I am confused as to where this script is run from when I put it under General Tab. My script file only contains the following line just for test (no .wav file movement yet; I trying to prove the concept for now):

touch success.txt

Once recording is done I see this in CLI:

== End MixMonitor Recording SIP/602-0857c070
== Executing [/var/spool/asterisk/monitor/test/script]

And that's it. It's stuck on the last line for ever. When I go to see if a success.txt was created, I don't see it meaning the script file didn't run. I also tried the following variation with no luck:

!touch success.txt

The file "script" has been given "chmod +x" already and ./script yields the file success.txt in the directory but it just doesn't do the job when run from FreePBX.
 

dicko

Joined
Oct 24, 2008
Messages
4,099
Likes
0
Points
0
#13
The script (as is all asterisk in Elastix) is run by the user asterisk
The asterisk user's home directory is /var/lib/asterisk
check there if you have un-anchored paths in your scripts

you need to anchor your paths and make sure they are writable by asterisk (the user)

if /var/spool/asterisk/monitor/test/script (the script that is called by the recording context) exists I suggest you replace touch success.txt with
touch /var/spool/asterisk/monitor/success.txt as a first step
 

torontob

Joined
May 18, 2008
Messages
219
Likes
0
Points
0
#14
Hmmm....I can't seem to work it. I still get the same result despite putting the script file in /var/lib/asterisk/ folder or defining full paths. Maybe it's the Asterisk user. How can I give permissions and where should I give permissions? Also, in General Tab I put full path for the script and I also put just script in the field and all of them just go:

Executing (/var/lib/asterisk/script)
or
Executing (/script)

and that's it.
 

dicko

Joined
Oct 24, 2008
Messages
4,099
Likes
0
Points
0
#15
Torontob:

I'm sorry you're having difficulties, but it does work as documented, perhaps a little time spent with a good linux tutorial and a full read of "cmd monitor" at voip-info.org and the digium lists will be productive.

Baby steps, start with
echo "torontob wants ^{MIXMON_DIR}^{CALLFILENAME}.^{MIXMON_FORMAT} that started in ^{CHANNEL} to go somehwere else"
as the command to run, just to convince yourself it is working

weatch the asterisk CLI output, be aware that there is no terminal assigned to output anything from the script, so all debugging must be done in bash. (by default the shell that the asterisk user has)
When you have the "parameters" right you can copy the execute line from the CLI and run it in bash (as user asterisk) replacing echo with your script of course. until you succeed ( as long as the file remains where it is of course).
 

torontob

Joined
May 18, 2008
Messages
219
Likes
0
Points
0
#16
Thanks for all the input Dicko. As you thought it was an Asterisk privilage issue.

The following did the charm:

chown -R asterisk.asterisk scriptdirectory
chown -R asterisk.asterisk script
 

torontob

Joined
May 18, 2008
Messages
219
Likes
0
Points
0
#17
Now that it seems the script is running, I have the following questions hoping to waste less time on these using your expertise:

1- What is the easiest way to have the script generate a directory which is in format YearMonthDay. For example, 20090326. The script has to be wise to realise this format in order to be able to perform something like:

cp /var/spool/asterisk/monitor/^{CALLFILENAME} /var/spool/asterisk/monitor/${YearMonthDay}/

I tried to explore the CALLFILENAME variable to see if I can extract a YearMonthDay variable but it seems not possible. FreePBX uses the php file:

/var/lib/asterisk/agi-bin/recordingcheck

to generate the file name which is in format:

Inbound Calls (rcin):
$agi->verbose("CALLFILENAME={$timestamp}-{$uniqueid}");
$agi->set_variable("CALLFILENAME","{$timestamp}-{$uniqueid}");


Outbound Calls (rcout):
$agi->verbose("CALLFILENAME=OUT{$exten['data']}-{$timestamp}-{$uniqueid}");
$agi->set_variable("CALLFILENAME","OUT{$exten['data']}-{$timestamp}-{$uniqueid}");


Out exmaple is:
-- Executing [s@macro-record-enable:999] MixMonitor("SIP/602-0861a770", "20090326-112647-1238081207.4656.wav||/var/spool/asterisk/monitor/test/orgscript") in new stack

I also tried to use hwclock and clock in linux but the format can't be segregated into YearMonthDay.

2- As for the variables go, I tried the following in the script and all didn't out the Call File Name to my txt file:

echo "Call file name:" ^{CALLFILENAME} > txt
echo "Call file name:" ${CALLFILENAME} > txt
echo "Call file name: ^{CALLFILENAME}" > txt


What could be wrong in this case?

Thanks
P.S. Hope I haven't given you a headache by now :)
 

ramoncio

Joined
May 12, 2010
Messages
1,663
Likes
0
Points
0
#18
filename-`date +%Y.%m.%d.%Hh.%mm`.wav
 

dicko

Joined
Oct 24, 2008
Messages
4,099
Likes
0
Points
0
#19
The callfile name:
20090326-112647-1238081207.4656.wav
has the date embedded in itself

echo $CALLFILENAME|cut -d '-' -f 1
will extract it

To be safe you should probably try to "mkdir -p" to each time before writing to it, that will suppress errors if the directory exists
 

dicko

Joined
Oct 24, 2008
Messages
4,099
Likes
0
Points
0
#20
??
echo "Call file name:" ^{CALLFILENAME} > txt
echo "Call file name:" ${CALLFILENAME} > txt
echo "Call file name: ^{CALLFILENAME}" > txt

the parameters used in calling the script will be accessible as $1 $2 etc. in bash.


#!/bin/bash
OUTDIR=`echo $1|cut -d '-' -f1`
mkdir -p /var/spool/asterisk/monitor/$OUTDIR/
cp(mv?) /var/spool/asterisk/monitor/$1 /var/spool/asterisk/monitor/$OUTDIR/
 

Members online

No members online now.

Latest posts

Forum statistics

Threads
30,916
Messages
130,922
Members
17,598
Latest member
giornaso
Top