Text Categorization

Text Categorization Framework Released

2012-02-10T01:53:00.000-08:00

Clarabridge the text analytics company recently launched Link, a framework of web services that gives direct access to their text analytics engine and their sentiment engine.

The Link engine can detect attitudes and emotions, identify key topics and phrases for text categorization.

Link has more than 15 categorization templates, plus natural language processing capabilities.

Chinese Blue Jeans Sweatshop the Subject of Expose'

2007-01-28T12:34:00.000-08:00

How would you feel if you knew that the jeans you're wearing were made by young Asian girls who got paid the equivalent of about six cents an hour? Not only six cents per hour, but under terrible conditions - they are forced to stay awake and working. And most of us are benefiting from this directly.

read more | digg story

The State of the Union: An Ordinary Citizen Responds

2007-01-22T21:23:00.000-08:00

It is imperative that our government recognize the fact that we the people are FED UP with the attempt to undermine American sovereignty, homeland security, and culture.

read more | digg story

Soldiers Work to Provide Iraqis with More Water Treatment Plants

2006-04-18T23:53:00.000-07:00

Soldiers Work to Provide Iraqis with More Water Treatment Plants
The progress comes none too soon for many of the residents of this northern Iraqi province. Some have never had drinking water available to their homes said Spc. Vilhelm Heerup, a civil affairs specialist with Company C.

Independent Journalist Bill Putnam Prepares to Visit Kurdistan

2006-04-09T11:00:00.000-07:00

Independent US journalist preparing to go to Kurdish area of Iraq, needs help to visit this dangerous area for two weeks to see what's really going on there..

read more | digg story

Jillian Ann's Diary - an Intimate Portrayal of Independent Music

2006-04-04T00:20:00.000-07:00

Jillian Ann, web icon, fashion model and Indie musician, has just released what may be the first 'real' show designed for Internet distribution entitled "Jillian Ann's Diary." Brilliant album, great video diary.

read more | digg story

Google To Telcos: Who Needs You?

2006-02-07T22:26:00.000-08:00

Telcos like AT&T and Verizon have announced their plans to extort money from Google and other sites if those sites want to get adequate bandwidth. But there's some evidence that Google may be planning to bypass the Telcos altogether, and roll out its own national broadband service.

My take on it is thta Google is building its own network for its own use, not to become an ISP. The ISP business is not as profitable as the other business Google does and google has no expertise in that area. Unless it buys a company, Google would be asking for big trouble if they did that, IMO.

So the google network I see is for their own use only.

read more | digg story

Russia: Spy gadget hailed as 'technological miracle' Worth tens of millions

2006-01-26T22:58:00.000-08:00

The 'rock' found in Russia that was being used by British agents to transmit data is said to be space-age technology worth tens of millions of dollars.

The FSB accused four British diplomats of involvement in a spy ring in which agents allegedly passed secrets through the device, located in a Moscow park.

The Russians said the British 'spy rock' is a multi-million dollar miracle of technology
http://newsblaze.com/story/20060126201650tsop.nb/newsblaze/TOPSTORY/Top-Story.html

read more | digg story

Dead Body Guy to Star in 'Horrorween' - a triumph of technology

2006-01-21T13:23:00.000-08:00

The internet, websites, blogs and ebooks made this happen.

A guy dreams about being on the TV and in the movies. He finds a book that gives him the courage to do something about it. He creates a website and a blog. Finds a great publicist and the rest is history - Newspaper, Radio and TV interviews and now - OFFERS !

read more | digg story

Hi-Res Pluto Liftoff

2006-01-20T11:14:00.000-08:00

Hi-Res image of Liftoff of the Atlas V carrying NASA's New Horizons spacecraft to a distant date with Pluto!

Its a great photo and an interesting project because pluto is too hard to study from here.

read more | digg story

Consumers Rejection of RFID Passports Consistent With Research Findings

2005-08-20T15:57:00.000-07:00

Consumers Rejection of RFID Passports Consistent With Research Findings: "RFID (Radio Frequency Identification) used to be an acronym known only by a handful of technology companies and retailers looking to implement slick new supply chain solutions, but it is increasingly becoming a consumer household word. Awareness of RFID among consumers has grown from just 28% in September 2004 to over 40% in March 2005. (Source: RFID Consumer Buzz March 2005)"

Perl books

2005-03-26T17:31:00.000-08:00

Perl books

My Text Categorizer written in Perl is doing a reasonable job, but I want it to do more. I'm finding that it misses many possible categories.

What I'm really looking for is natural language processing, but with some differences. I want it to know about paragraphs of text and I want it to be able to handle "must have" and "must not have". I already built those two features into the existing text categorizer, but they only work if the categorizer selected the document for a category.

The problem is that its missing some categories.

I will need to text it, but maybe I just want a boolean learner.

The main problem with what I'm doing is that I don't want to have a large set of documents that act as the training set. All I want to do is to have a slim specification of the matching text, like a set of search engine keywords.

Maybe I'll look for someone who can help me get it done.

O'Reilly Network: What Percentage of Developer Positions Should Be Junior?

2004-08-08T18:24:00.000-07:00

O'Reilly Network: What Percentage of Developer Positions Should Be Junior?
by William Grosso -- I've been paying attention to the local help-wanteds for the past couple of weeks. By my count, approximately 4% of the positions being advertised are junior positions. I wonder what that means?

You would think it would be more than 5%, but there are some special circumstances right now.

First, there are some experienced developers out of work, who are probably willing to accept lower wages and an employer isn't going to train someone when they can get experience at a good rate.

Second, employers are learning they can do more with less.

Third, some employers think they can save money by outsourcing overseas.

Fourth, the economy isn't exactly stable and employers are holding off.

Fifth, some companies are on the edge of crashing and they are not only not hiring, but they are going to create some unemployed developers soon.

The number of companies with vision, foresight and cash is not a big number. And for some reason or other, there are a lot of Venture Capitalists who can't even give their money way. That's not to say there aren't companies willing to burn up their money - I think it means the VC's aren't being presented with enough business plans that make sense.

TrackBack URL to William's article:
http://www.oreillynet.com/cs/user/trackback/cs_msg?x-lr=cs_disc/9427&x-lr2=wlg/5388&x-a=submit&trackback=1

Categorization Improvement

2004-06-12T15:15:00.000-07:00

After another few days effort and many experiments, a reasonable improvement that I am very happy with.

The latest system uses Naive Bayes Categorization with a low threshold, that gives more positive hits and more overage, coupled with 3 verifiers that enhance the results in various ways.
It even discovered 30 real hits that I missed in my manual classification.

The Boolean Natural Language NOT verifier reduces overage

The Boolean Natural Language MUSTHAVE verifier

The Composite Classifier increases hits by triggering a categorization when one category makes up part of another, for instance Chile is a South American country and the Composite classifier adds any story tagged for Chile into the South America topic, if the standard South America topic didn't already tag it.

Results

This is the result without verification
379=Hit, 10=NoCat, 172=OverCat

The NOT verifier reduces overage, reduces hits
372=Hit, 16=NoCat, 155=OverCat

The MUSTHAVE verifier greatly reduces overage, reduces hits
349=Hit, 37=NoCat, 11=OverCat

Together, the verifiers cause a further improvement
342=Hit, 44=NoCat, 3=OverCat

Finally, adding the Composite Classifier increases hits
359=Hit, 44=NoCat, 4=OverCat

Overall, its a good improvement. Out of 600 manually tagged topics, the Categorizer picks up just over half of them and the overage is only 4 topics. This is very good.

Now I'm ready to test documents from other sources.

I suspect that further improvements will be gained by improving the category definitions.

I still want to try the SVM (Support Vector Machine), as soon as I get a new server set up.

Eureka! I found it!

2004-06-07T21:39:00.000-07:00

Now I'm happy.
I have more hits and less overage. In fact a lot less overage.
When I first set up the changes, there were about 6 overages and I thought that was really great.

Then I realised that they weren't overages at all - the auto
categorizer actually picked up topics that I missed.

So here are the results.
t7bbxcNLQ1: 230=Hit, 54=NoCat, 3=OverCat

There should be only 10 with no categories, so there is still room for improvement.

One improvement was to better specify some of the categories.

What brought these improvements was using NaiveBayes with a low threshold of 0.1 instead of 0.3 and then using a boolean/Natural Language query on the results.

More information soon.

Experimenting with front_bias

2004-05-30T23:41:00.000-07:00

I have manually identified 213 categories in the small corpus
of 220 documents

# front_bias => 0.9
perl Z-Categorizer.pl stopwords7 b
93=Hit, 106=NoCategory, 35=OverCat

front_bias => 0.9
perl Z-Categorizer.pl stopwords7 b
175=Hit, 51=NoCategory, 40=OverCat

front_bias => 0.99 *** Best result ***
perl Z-Categorizer.pl stopwords7 b
176=Hit, 47=NoCategory, 37=OverCat

front_bias => 0.9999
perl Z-Categorizer.pl stopwords7 b
154=Hit, 52=NoCategory, 40=OverCat

front_bias => 0.99,
stemming => "porter",
151=Hit, 22=NoCategory, 81=OverCat

Handling the NOT case in Text Categorization

2004-05-30T16:53:00.000-07:00

I'm not sure that this is the best way to handle it, but as I havent discovered a better way so far, this is my solution:

I've statically created an extra topic that is teh same name as teh topic I'm looking for, except that it has a minus sign in front of it. This topic contains the items that I don't want in the topic, thus qualifying the input document.

For instance the text for the JORDAN category is "Jordan" and the text for the -JORDAN category id Michael. Then on output, I take 2 runs through the corpus list. The first one gathers the NOT cases, remembering only the category name, with the '-' character removed.
In the second loop, if the topic is in the NOT has, then it is skipped.

This is not the ultimate in elegance, but it works and that's all I care about right now - and its only a few lines of code.

I could handle this by using my original topic data structure in which there was a NOT entry in the body of the topic definition and then splitting it out at the time the base knowledge set is created.

So, by using just one NOT case, the hit/miss score looks like this:

t4bbxcT8: 179=Hit, 43=NoCat, 38=OverCat

An improvement over the previous best, which was:

t7bbxcT7: 182=Hit, 39=NoCat, 42=OverCat

What are the Text Categorization overages

2004-05-30T00:32:00.000-07:00

The overages are interesting. They are all related to miscategorization due to acting on a single word rather than a phrase and my inability to program NOT into the system.

for example, a Horoscope story gets picked up by the Cancer
topic. I could stop that with NOT horoscope, aquarius etc

A Michael Jordan story is tagged by the Jordan (country) topic
2 Appeals court stories are tagged by the Supreme Court topic

A child custody story is tagged by the Child Abuse topic

I was hoping to find other people working on this problem but I can't find anything recent.

At least I'm getting closer to where I want to be, but I'm now over a week behind.

Hacking my topic definitions and rechecking manual classifications

2004-05-29T23:57:00.000-07:00

After almost a whole day of trying to get individual term weighting running, I decided to have a break and try something else.

First, I saved the topic definitions and started hacking them.
This classification system has a few problems. First, it only knows about words, not phrases, so Michael Jordan and Jordan the country get classified the same.

I obviously need some Word Sense Disambiguation here, but I don't know how to do that with these modules. So I settled on hacking the topic definitions.

At about the same time, I reviewed the hits and misses and realised that I may have missed classifying some topics in a few documents.

I'm pleased with the results of this effort because now, the number of positive hits is up and the overage is down. Its moving in the direction I wanted. Also, the number of files with no category is down.

t7bbxcT1: 155=Hit, 35=NoCat, 82=OverCat
t7bbxcT2: 159=Hit, 36=NoCat, 76=OverCat
t7bbxcT3: 160=Hit, 36=NoCat, 75=OverCat
t7bbxcT4: 160=Hit, 36=NoCat, 76=OverCat
t7bbxcT5: 162=Hit, 36=NoCat, 73=OverCat
t7bbxcT6: 176=Hit, 36=NoCat, 51=OverCat
t7bbxcT7: 182=Hit, 39=NoCat, 42=OverCat

So that was a very useful exercise.
I tested KNN and Rocchio again.

t4kbxcT7: 44=Hit, 174=NoCat, 3=OverCat
t4rbxcT7: 179=Hit, 84=NoCat, 65=OverCat

Also tested term weighting
t4btxcT7: 180=Hit, 40=NoCat, 52=OverCat

So binary weighting with cosing smoothing and lots of
stopwords with a Naiive Bayes Learner works best so far.

Handling the NOTs should definitely do it for me.

Attempts to improve the Over-classification

2004-05-29T15:25:00.000-07:00

The first thing I tried was to alter the term weighting system.
The best results came from using bxc. I was already using bxx in the previous tests. The difference is cosine smoothing of results. This reduced the overage and increased the number of good hits.

tfidf_weighting specifies how document word counts should be converted to vector values.
This scheme uses the three-character specification strings from Salton & Buckley's paper "Term-weighting approaches in automatic text retrieval". The three characters indicate the three factors that will be multiplied for each feature to find the final vector value for that feature. The default weighting is "xxx".

The first character is "term frequency".
The second character is "collection frequency".
The third character is "normalization"

b Binary weighting - 1 for terms present, 0 for terms absent
x No change - multiply by 1
c cosine normalization - multiply by 1/length(doc_vector)

t7bbxc: 124=Partial, 157=Hit, 35=NoCategory, 62=OverCategory, 84=TotalOver
t7rbxc: 81=Partial, 161=Hit, 83=NoCategory, 57=OverCategory, 92=TotalOver
t7kbxc: 39=Partial, 39=Hit, 178=NoCategory, 4=OverCategory, 4=TotalOver

Naiive Bayes, KNN and Rocchio all improved with smoothing and surprisingly, Rocchio came right up with Naiive Bayes and got the best number of positives and only slightly more negatives.

I'm trying to get a server set up where I can run a C compiler and build a Support Vector Machine (SVM) because I think this would be a very interesting comparison.

The next thing I tried was to reduce the amount of document data passed to the classifier. I tried sending it only the document TITLE and the 200 character summary.

I call this the shortdoc test. As you see, it reduced the overage, but the number of good hits is down too
shortdoc: 39=Partial, 39=Hit, 168=NoCategory, 14=OverCategory, 15=TotalOver

The next thing to do (apart from building the SVM) is to try individual term weighting in a category.
If I could do that, I'd be able to to get the NOT terms working, which I think are going to give the biggest improvement. There won't be more hits, but there should be much fewer overages, which is what I need right now.

First Categorization Results (Numeric)

2004-05-25T09:07:00.000-07:00

These are the first results that will hopefully lead me to making changes in various areas to improve the automated categorization.

k = KNN
r = Rocchio
b = Naiive Bayes

t1k: Exact=4, Partial=22, NoCategory=205, OverCategory=1
t2k: Exact=11, Partial=38, NoCategory=187, OverCategory=4
t3k: Exact=14, Partial=54, NoCategory=177, OverCategory=3
t4k: Exact=16, Partial=58, NoCategory=173, OverCategory=3

t1r: Exact=5, Partial=28, NoCategory=22, OverCategory=147
t1rn: Exact=7, Partial=30, NoCategory=28, OverCategory=171
t2r: Exact=8, Partial=32, NoCategory=43, OverCategory=78
t2rn: Exact=15, Partial=54, NoCategory=64, OverCategory=115
t3r: Exact=21, Partial=86, NoCategory=90, OverCategory=67
t3rn: Exact=21, Partial=86, NoCategory=90, OverCategory=67
t4r: Exact=10, Partial=28, NoCategory=30, OverCategory=22
t4rn: Exact=22, Partial=90, NoCategory=92, OverCategory=62

t1b: Exact=22, Partial=74, NoCategory=3, OverCategory=159
t1bn: Exact=22, Partial=74, NoCategory=3, OverCategory=159
t2b: Exact=34, Partial=92, NoCategory=17, OverCategory=124
t2bn: Exact=34, Partial=92, NoCategory=17, OverCategory=124
t3b: Exact=45, Partial=108, NoCategory=22, OverCategory=100
t3bn: Exact=45, Partial=108, NoCategory=22, OverCategory=100
t4b: Exact=49, Partial=104, NoCategory=24, OverCategory=96
t4bn: Exact=49, Partial=104, NoCategory=24, OverCategory=96

Discussion of these results:

Overall

It is obvious that increasing the number (and probably quality)
of stopwords leads to better categorization, so that is an area to be investigated further.

KNN

KNN gives the least number of overcategorizations, but it obviously misses a lot of things too

Text Categorization Algorithms and Methodology

2004-05-24T22:52:00.000-07:00

In this first test run, I have 4 Algorithms for selecting Article Categories.

Guesser, Naiive Bayes, KNN and Rocchio
I will use these abbreviations: g=guesser b=bayes k=knn r=rocchio

There are other interesting algorithms I want to try, but due to
the limitation of no access to a C compiler on this server, I
couldn't build a Support Vector Machine (SVM) or a Decision Tree
(DT). I will try these later.

k-Nearest Neighbours (KNN) is an example-based classifier and
Rocchio is a linear classifier, which depend on a test set of
documents, so these may no be useful for what I want.

Guesser was by far the fastest to execute, but now that I have a baseline, its easy to see why - its completely useless.

With a base set of 224 documents, using 4 different sets of stopwords and stemming or no stemming, there were no exact matches and no partial matches.

The test names tell something about the test - they are also
the filenames I used to store the results.

t = test
1-4 = the stopword files = 5, 23, 571 and 1602 words
(I'll explain these in detail later)
g,b,k,r = the algorithm used, as shown above
s or n = stemming or no stemming

Im not going to discuss results in this post, except to say
that I'm not going to use the Guesser algorithm again
because its a waste of time - OK for academic interest, but
no good to me.

This is how the results will show. These are the guesser
results - you can see why its a waste of time.

The table shows the number of Exact matches, Partial matches
where the Algorithm gave a subset of the manual categorization
No category at all and finally, categorization, but additional
topics. These will need further investigation.

I would rather have it under-categorizing than give extra
categories that are false. For instance, I'd rather it not
put a Washington State story in the Washington DC category.

However, there could be perfectly valid over-categorization that
I missed seeing, so this first set of results will need more
investigation.

These are the results from Guesser

t1gs: Exact=0, Partial=0, NoCategory=82, OverCategory=139
t1gn: Exact=0, Partial=1, NoCategory=86, OverCategory=134
t2gs: Exact=0, Partial=0, NoCategory=76, OverCategory=145
t2gn: Exact=0, Partial=0, NoCategory=75, OverCategory=146
t3gs: Exact=0, Partial=0, NoCategory=74, OverCategory=147
t3gn: Exact=0, Partial=0, NoCategory=76, OverCategory=145
t4gs: Exact=0, Partial=0, NoCategory=82, OverCategory=139
t4gn: Exact=0, Partial=0, NoCategory=85, OverCategory=136

Text Categorization baseline

2004-05-24T21:03:00.000-07:00

I spent 4 hours manually categorizing almost 400 documents, to use as the baseline for the automated tests.

After that I ran a quick check to see how close any of the automated tests were to the baseline. some of the automated tests were completely useless, while others were remarkably close.

One big problem I noticed easily was the confusion between Washington DC and Washington State and there was one instance of George Washington.

The Categorizer Algorithms all seem to only handle the documents word by word, less the stopwords and any stemming.

Here are a list of things I think it needs to handle:
- 2 and 3 word phrases
- proximity (x within 2/3/4 words of y)
- NOT a word / phrase / proximity phrase
- a OR b OR c OR d etc
- MUST contain a OR b OR c etc AND d OR e OR f etc

If I can get all this into the system, I'll be very happy.

The next report will show initial results comparisons between the Human Categorizer (me) and the automated Categorizers.

Text Categorization improvements

2004-05-23T10:42:00.000-07:00

Well, back to the drawing board. none of those searches helped.
It was basically a waste of 4 hours.

So I went back to AI::Categorizer and found a really good paper by Fabrizio Sebastiani called Machine Learning in Automated Text Categorization.

I also thought through what I was trying to achieve and realised I need to build an extra module of my own and also need to call the Categorizer differently.
- Its amazing the things you can accomplish when you think and investigate before you act!

So I revisited all the documentation (its not set out to be easily accessible), reworked the code and came up with results much closer to what I was looking for.

I ran some tests and got interesting results, which I'll publish soon. The poor initial results were what made me look elsewhere, but now I realise that subtle changes can drastically improve the results.
Now its back to the code and tests.

Text categorization

2004-05-22T10:39:00.000-07:00

I've been experimenting with text categorization for a new project. I started with many searches on Google. There are quite a few commercial products from large companies, all with large price tags. Probably all worth it, because its not a simple thing to do.

I was only experimenting so I was looking for code to build on. Ken Williams' AI::Categorizer is a perl module that keeps coming up in the searches.

After a day of searching, I decided I'd try it. Neat interface, easy to follow the examples. Unfortunately, I got really weird results. It must be because of the way the Learner is configured. The winning pattern seems to dominate all the others. I was tryiing to run it on my site for Perl Books.

Unfortunately, although I have SSH access, I can't run the C compiler and that meant I could only try 3 of the basic learners. The Decision Tree learner has a lot of C code and it needs to be compiled and linked to the perl.

After another day of testing, I was looking in dmoz and I saw a link to Text Categorizer, so I'm going to investifate that.

Business Strategy

2004-05-18T18:23:00.000-07:00

If your small business isn't performing, you probably don't have a strategy to make it perform and keep it on track.

At least not a written strategy.
Its a mazing that many business just fly by the seat of their pants.
Some of them probably do OK but I can't help thinking that they could do much better if they had a roadmap.

Its like driving in the country without a map. I did this last week, north of Lincoln. Just kept moving south and west and eventually found a main road. Then I looked at the map when I got home to see where I should have gone and realised I could have been home an hour earlier. I'll remember to take the map next time!

Last month, I ran a series of PPC tests on Overture advertising Small Business I was surprised at the number of clickthroughs. Most of them were for "free business plan".

I'm not sure whether people are looking for a plan to use or whether they need some guidance. In any case, without a strategy, what's the point?

I tried to tune out the freebie seekers by setting the ad up to dissuade them. This month I'm going to try 2 ads in the same section. One will dissuade them and the other won't.

Look here for the results in a few weeks.

Alan.