Converting XML

[constantin01]

Hi

There is xml-file (350 mb) that is a dictionary of tagged russian words. I want use it to tag words but problem is that search in xml-file is very time-complex thing.

My question is: in what I should convert xml to handle this data more efficiently? I think of trie: whether it is good idea to create "forest" of tries ? Each word will look like this :



// "КОТ" - this is a russian word for a male cat

[Gribouillis]

it is good idea to create "forest" of tries ?

I don't think it's a very efficient idea. The 350 mb can probably be reduced drastically. How many words are there in the dictionary? how many different tags? How many tags per word? Are there other informations?

[constantin01]

it is good idea to create "forest" of tries ?

I don't think it's a very efficient idea. The 350 mb can probably be reduced drastically. How many words are there in the dictionary? how many different tags? How many tags per word? Are there other informations?

It contains about 150.000 words (lemmes) and each word has 7-11 forms; each form has 4-6 tags.

[DeaD_EyE]

lxml should handle this data.
It's written in C and very fast. It has the same api like the built-in module xml provides.

[Gribouillis]

I made an experiment of creating a sorted list of 150000 * 10 unique random unicode words having between 5 and 12 unicode characters. By pickling this list I obtain a 50 megabytes file. By loading the whole list and using bisect.bisect() to find the word, it takes 0.4 microseconds to find a word in the list (according to module timeit). You could then create another list containing for each word a reference to the tags (for exemple a python tuple of integers or a shorter structure such as a byte string containing the indices of the tags). This should give a fairly efficient way to find the tags, at the cost of some RAM consumption. A python dictionary could perhaps be used as well.