Slashdot Log In
Linguistics Meets Linux: A Review of Morphix-NLP
Posted by
CowboyNeal
on Thu Dec 11, 2003 09:24 PM
from the natural-linux-processing dept.
from the natural-linux-processing dept.
Emre Sevinc writes "Zhang Le, a Chinese scientist working on Natural Language Processing has decided to pack the most important language analysis and processing applications into a single bootable CD: Morphix-NLP. More than 640 MB of NLP specific software is included and there's still a lot of place on the CD which uses a compressed filesystem for bringing us the best of both worlds."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Ironic.. (Score:5, Funny)
All this language processing packed onto a single CD yet
Re:My wish (Score:2)
You do know this is /., right?
Noooo (Score:5, Funny)
Re:Noooo (Score:2, Informative)
Re:Noooo (Score:2)
that's pretty cool (Score:3, Insightful)
Neat.
--dw
It's actually useless for that (Score:5, Interesting)
Actually, this software seems like it would totally useless for that purpose. The software was developed and has a bunch of heuristics and domain knowledge put in by experts in english or the relevant language. Without similar expertise, the software can't be adapted to a new language. The software isn't a universal translator.
So your hypothetical anthropologists or translators would still need to spend time and learn the language in question.
Parent
Re:It's actually useless for that (Score:4, Informative)
At least half the tools are general purpose applications for constructing various kinds of models, whether they be trees or HMMs or n-gram models or entropy models.
Believe it or not a lot of NLP work gets done on understanding algorithms that apply broadly across languages.
There is some English specific stuff on the CD, but most of it isn't.
The only software
Parent
Re:that's pretty cool (Score:5, Interesting)
Actually, not very many anthropologists these days do much linguistic work. That's partly because linguistics has developed as a separate field and partly because cultural anthropology was largely taken over by Postmodernists, as a result of which it has nearly died. Most research on "exotic" languages these days is done either by linguists or by missionaries (who want to translate the New Testament).
I am a linguist and have done extensive fieldwork, mostly on Carrier [ydli.org], the native language of a large region of northern British Columbia. (I also hack a little. Once upon a time I wrote the head-final shell mentioned in Charles Dodgson's comment [slashdot.org].) Software is increasingly used for this kind of work, but for the most part it is not the sort of NLP software provided on the Morphix-NLP CD. A lot of that software is useful primarily if you've got a large corpus to work with, and it often presupposes that some basic resources exist, such as a lexicon, or at least a wordlist with part of speech information. For many languages even basic resources such as a lexicon don't exist or aren't available in electronic form, and when you're dealing with really small languages, there aren't any ready-made corpora, such as news text. If you want a text corpus, you've got to make it yourself, usually by recording people telling stories or whatever, and transcribing it. This is an important part of fieldwork, but its incredibly slow and tedious.
There are some tools designed specifically for this kind of linguistic research. One is Transcriber [upenn.edu], a tool that assists a human being in transcribing audio recordings. One of the older tools is Shoebox [sil.org] a dictionary database program for field linguists, originally written to run under DOS.
Some of us have used Unix tools to extract and process information, e.g. grep to do regular expression searches. Ken Church at Bell Labs used to give a tutorial "Unix for Poets" on how to use Unix tools for linguistics. Here is his handout [att.com]. For example, I've produced dictionaries of several dialects of Carrier using scripts written mostly in AWK plus the usual Unix tools, controlled by elaborate Makefiles. Some of us also use emacs a lot, not only as an editor but for doing searches. If you're interested in what kinds of software are of interest to linguists, you might check out the Computational Resources for Linguistic Research [upenn.edu] page.
It is worth mentioning that spread of the internet has made available a lot of useful material for linguistic research. There are now quite a few languages for which you can obtain a good chunk of text (say at least 100K words), and often you can find parallel text (that is, the language you're interested in plus a translation into English or another language that is useful to you). But this works mostly for relatively big languages, that is, say, languages with a million or more speakers. There are around 340 such languages, depending on how you count, about 2% of the world's oral languages.
One topic that concerns some of us is how software and other technology can speed up the process of documenting dying languages. Languages are rapidly become extinct - some experts estimate that as many as 90% of the languages currently spoken will be extinct in 100 years. [Computer languages may be proliferating at the same rate.:)] The late Ken Hale [anu.edu.au] had seven languages die on him. If we don't find a way to speed up the documentation, or slow down the rate of extinction, most of those languages are going to die without very much being known about them.
Parent
Great... (Score:2, Funny)
Re:Great... (Score:5, Funny)
Parent
Re:Great... (Score:4, Interesting)
I guess this would interest you too. BTW, have you read "Le Ton Beau de Marot" by Hofstadter?
In 1977, Xerox adopted Systran for internal translations by creating a Multinational Customized English that's easier to translate. [1]
In 1930, C.K. Ogden proposed a tiny version of English: just 850 words that could be learned in a few months and used to say anything. He called it Basic English (BE). [2] [3]
Parent
Re:Great... (Score:2)
Re:Great... (Score:3, Interesting)
However, your BNF grammer is likely to come unstuck as soon as you try to parse either casual english or moderately complex english. Either one very quickly leads to adding lots of infrequently used grammar rules, and hence lots of ambiguity in even simple sentences.
The
So this means (Score:3, Funny)
Anyone remember Forum 2000? (Score:3, Interesting)
Re:Anyone remember Forum 2000? (Score:4, Informative)
http://www.forum2010.org/
Parent
Re:Anyone remember Forum 2000? (Score:2)
For those who aren't surfing at 0 or -1, someone graciously provided this link [forum2010.org].
Now I'm going to surf and hope I find the wisdom of Ayn Rand on the new site as well. *cackles*
Re:Anyone remember Forum 2000? (Score:4, Informative)
Forum 2000 and 3000 died mainly because the people who ran them got bored and/or wanted to work on their graduate theses. It sure was fun to play with the Zephyr interface while it lasted, though.
I wonder whether Forum 2010 is run by the same folks. I doubt it since Forum 2000 and 3000 were both Carnegie Mellon projects, and forum2010.org is registered to someone in St. Louis.
Parent
Re:Anyone remember Forum 2000? (Score:2)
Forum2000 is dead. Long live Forum 2010! (Score:4, Informative)
That's me, actually. You can't expect hundreds slashdot geeks suddenly slamming my site and having me not notice. ];-)
Forum 2010 had, in fact, nothing to do with the great fellows at Forum2k/3k aside from inspiration. And, just to end the rumors, I built the F2.01k matrix and all my own SOMADs as a senior project for my Comp Sci degree at Fontbonne University [fontbonne.edu].
Now, I'm late for a date! Please don't destroy the matrix while I'm gone!
Parent
Re:Forum2000 is dead. Long live Forum 2010! (Score:2)
Why Linux is great for doing applied linguistics? (Score:5, Informative)
This page [bigpond.com] has some reasons.
Download Link (Score:4, Informative)
Try not to kill their site. If someone has downloaded it, it would be nice of them to post a
Chomsky and stuff (Score:2, Interesting)
God damn, them are some fancy-schmancy sounding titles! Does anybody ever get the feeling sometimes that maybe things are simpler than our smartest people currently make them out to be? If you can't talk as simple as I'm talking now, you ain't really "nailed it."
The reason I think this is true: back when all mathematicia
Re:Chomsky and stuff (Score:2, Insightful)
What you're seeing here is the process by which that happens. Chomsky especially is someone whom I don't consider to want to "make [things] out" to be more complicated than they are; on the contrary, he seems to be more about wanting to understand the *true* process that is at work, not the pre-accepted soci
Re:Chomsky and stuff (Score:5, Interesting)
Linguists don't think Knuth is very lucid.
Linguistics is neat. Syntax (the study of the structure of language), Phonology (the study of the interactions of sounds and what a child has to actually 'learn'), Phonetics (the study of the human language system and the sounds that it can produce/hear), and Morphology (the study of the smallest possible unit that holds 'meaning') all work together to form an idea of what goes on in the human mind.
Parent
Re:Chomsky and stuff (Score:2)
I still find it hard to believe the original parent was serious, though... Roman numerals... :)
Re:Chomsky and stuff (Score:4, Informative)
He pretty much defined linguistic theory for the past 40 years. Once he had a voice he turned into somewhat of a political critic. A conspiracy-theorist. I don't see him solving any political problems, and I don't know how well respected he is by those who study such things, but I think he's a loon. (But, oh god, I wish I could study with him.
Chomsky's papers are tough to comprehend for beginners. (Which I am.) Those who are interested in learning Chomskian theory may wish to pick up some Andrew Radford. (he is very understandable, and his book "Transformational Grammar" is aimed at the undergraduate level syntax class. Once you tackle that, you can read Haegemann, "Government and Binding," which seems to be the most used graduate level book... but this one is quite boring.)
In the meantime, a linguistic glossary which may help you get through some of the papers you may find: http://tristram.let.uu.nl/UiL-OTS/Lexicon/
Parent
Re:Chomsky and stuff (Score:2)
Especially considering that 3, 7 and 12 were all 3 digit numbers, whereas 2, 6, and 9 had 2 digits, and 1, 5 and 10 had one; and 8 had four! Holy crap!
This has to be the funniest troll I've read in ages. My compliments!
Re:Chomsky and stuff (Score:5, Interesting)
First of all, most practical NLP techniques aren't *that* complicated simply because they must be able to be computed quickly. There are quite a few statistical hacks prevalent
Most NLP techniques use probabilistic variants of two models finite automata and pushdown automata (both models are actually pretty simple, but if you don't know what they are, they may sound complicated).
Finite automata consume input and transition to different states (a finite number of them) based on that input. They can also be interpretted as generating output instead of consuming input.
Push down automata are almost the same except that they have a stack that they can push symbols onto. Another name for push down automata are Context Free Grammars.
As I said above, most NLP techniques use probabilistic variants of and small extensions to these two concepts.
The reason that Markov models (probabilistic finite automata) work so well to model speech is because they are flexible, simple, and linear just like speech. The reason that CFGs work so well to model language is that they are flexible, and hierarchical, and so can capture the recursive nature of language (think about "the man who killed the horse who killed the dog who...").
Having said all of that, I don't think that these models capture the way that humans process language/speech. I think that neural networks have the potential to capture this better. They just aren't mature enough. We also don't really have a good architecture to run neural networks. A human brain has about 10^14 neurons (within a couple of orders of magnitude) that run in parallel. Try simulating that on todays serial architectures, and you'll run into problems.
So my hypothesis is that there is probably some inherently simple learning algorithm for neural networks that we just don't know yet that will help solve many different types of problems (there is some biological evidence of there being a single learning algorithm implemented in the brain).
So yes, there is likely a simpler answer, but until we know it, we have to use heuristics and statistical hacks in order to build systems that work.
As to science in general, the reason it all sounds complicated is twofold:
First things interect in a very chaotic way. Even if the interactions are simple, when you compose many very small interactions, you find complex behavior.
Secondly, even if the interactions are actually simple, we humans with our Neutonian intuitions have a hard time understanding non-Neutonian interactions.
Hope that helped.
Parent
Re:Chomsky and stuff (Score:3, Informative)
Actually, Chomsky (or one of his contemporaries anyhow) discovered early on that almost no natural language can be represented solely by regular languages, or even context-free languages. Chomsky initially even tried to use unrestricted/semi-Thue grammars to represent natural languages, but realized just as quickly that this HUGE class of languages is much, much too big (in fact, it's actually Turing complete, and only useful to those doing research in the theory of computation, not the theory behind human
Do you honestly believe that? (Score:3, Interesting)
That's also why none of the good stuff was made by the Romans - it was the Greeks, then the Arabs that had good numerals, made the discoveries, before the knowledge of a proper number system finally retu
Re:Do you honestly believe that? (Score:2)
If you think that's ever going to be something you can slap up on the blackboard in an hour, you're wrong.
All I'm saying is that 2000 years ago it took a 60 year old man hundreds of pages to describe techniques for long division, and they had LONG, LONG discussions about how stuff was made of earth, wind, and fire . You *seriously* believe similar advancements won't be made 1000 years from now that put our science in a similar light?
I'm not saying the techniques 2000 years ago weren't valid, or the
Omission of Gate (Score:3, Informative)
Should it be patented? (Score:2)
Because if it can be, we have to secure it with something before a corporation patents it!
Re:Should it be patented? (Score:2)
It's called prior art.
Memories (Score:3, Interesting)
Natural languages useful for spam filters? (Score:3, Insightful)
any of these natural language tools
can be helpful for spam filtering?
Cheers, Joel
The base Morphix (Score:2, Informative)
For documentation, you may want to have a look at the Morphix Wiki [xs4all.nl].
There is a downside to Natural Language Processing (Score:2, Interesting)
As a simple example, take spell checking. When the computer can remember the spelling for every word and fix it automatically, who is going to worry about spelling simplification or reform? Yet changing to a standardized phonetic spelling would probably help people in the long run, if only by allowing children time to actually *write* rather than spending time in rote memorization and spelling bees
and there's still a lot of place on the CD (Score:2, Funny)
OK, I get that it's a Chinese scientist working on this, but it's about language. Should the Slashdot article really have been written in Chlinglish?
Random musings from an ex-linguist. (Score:5, Insightful)
Linguists have always been geeky. Don't forget that Larry Wall is a linguist first.
The only computer class I ever took was in 1983 called "Computer tools for natural language analysis". It was an introductory Unix course. We learned grep, awk, sed as well as tools like vi, Mail, and rogue. And a tiny little bit of C. But since then I've taught C at the graduate level.
Linguistics is all about the reprensentation and manipulation of information. But instead of it being about languages we design for particular purposes, it is about the language system that we use naturally.
Suppose you have a few thousand languages that you know were written with the same tools (like lex and yacc, but not lex and yacc), but you have no access to those tools. Suppose you are trying to figure out what those tools are from examining the languages (not the compilers) that have been specified using those tools. That is what theoretical linguistics is trying to do. We know that the specification of English and the specification of Dyirbal and every other human language out there are somehow "written" with the same tools. It's pretty need stuff.
Linguists were early adopters of TeX, have had a Unix affinity for a while, and as people who are interested in how information is internally represented and manipulated, like reading the source.
I remember once nagging the sys admins to always make sure that there is a man page for anything added to /usr/bin or /usr/local/bin.
The next day, they asked me to look at the manpage for something to see if it met with my approval. The DESCRIPTION was the C source. I was happy to say that it did, indeed, meet with my approval.
At one point, a well known professor (Geoffrey Pullum) had written a little essay for a newsletter on the "grammer of Unix" using linguistic style analyses of the shell. Naturally several of us feigned outrage at his confusion of "Unix" with the shell. Another linguist (Bill Poser), went so far as to write a shell which was verb (command) final, and post-positional. That is instead of saying
/bin/sh chsh
cat foo bar > bang
you would say
foo bar bang > cat
That is, the arguments preceed the command, and the redirect symbols go after the filename they redirect to or from. Now for various reasons, I had root access on a machine that Pullum used. So I changed his shell to this command final one. He actually caught on remarkably quickly. And after a quick
he was ready to concede the point.
For me, there is no surprise that linguists, and particularly computational linguists, are OSS enthusiasts. But that is enough of my random musings for now.
Re:Good Chinese Compression (Score:5, Funny)
It's the Japanese who has problems pronouncing L's... and the Chinese have problems pronouncing R's.
The Westerners on the other hand, can pronounce almost anything, but will never ever get facts right
Parent
Re:Good Chinese Compression (Score:2, Insightful)
English is the only language I know but I studied Mandarin chinese for a few years.
There are all sorts of things in there that we have a lot of trouble pronouncing.
actualy (Score:2, Funny)
stiff sodomy laws? theres a joke in there somewhere...
Re:How many of you really support OSS? (Score:2)
You should take a look at posts on Mozilla, KDE, and GNOME, and you will see that people do get behind OSS for reasons other than elitism.
S
Re:How many of you really support OSS? (Score:2)
Slashborging (Score:3, Funny)
Even though they're stupid as hell, I was beginning to miss them.
Re:How many of you really support OSS? (Score:2)