Faster data access code

[TrafficProducer]

Faster data access code

I am using PERL to search through a number of files.

These files are combined into a large, (25Mb or more) normal array which is parsed and searched on.

Here is the outline of the pseudocode used:-

Open INFILE
@data_1 = <INFILE>;
close INFILE

open INFILE..
@data_2 = <INFILE>;
close INFILE

@all_filed_bargain_data=
(
@data_1,
@data_2
);

Then

While(not eof)
{
Parse the data from each array element number to get string_to_look_in

look for each Keyword in the string_to_look_in using:-
if(lc($string_to_look_in) =~ m/\b$search_keyword\b/gi) # Any case Whole word found
{
$keyword_found=$TRUE;
}
}

This works quite well but I whish to improve the speed of the searching, it takes about 4 to 6 seconds to return the results using the code above.

Questions, which would be faster

a) MySQL
b) C or C++
c) Other language
d) Stay with PERL but use Hash associated arrays instead of plain arrays as used now.
e) Use perl-mod
f) An other ideas

Wesite in question:-
http://www.comparebargains.com

Thank you.

[redcircle]

you could use grep and inject it into perl and manage the data from there.

[TrafficProducer]

use grep and inject it into perl and manage the data from there.

OK, this is has been done.

I think this is better because I now do AND searches..

As for speed, I think this is about it works out about the same after the AND'ing of search words so I'm still looking for a speed boost.

Thank you for the Grep idea this has improved things.

ref: http://www.comparebargains.com

[TrafficProducer]

Speed now improved Thanks to Grep..

Another update.

By directly feeding the results of one Grep into another Grep to carry out the ANDing speed has been improved.

Thank you for the Grep idea.

[mushroom]

I beleave the MySql is the best solution, provided that your tables are built with your querys in mind. Speeds of a fraction of a second are common even with large data bases and complex querys, if the tables are optemized for the querys.