{"id":424,"date":"2012-10-29T07:52:18","date_gmt":"2012-10-29T06:52:18","guid":{"rendered":"http:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/"},"modified":"2012-10-29T07:52:18","modified_gmt":"2012-10-29T06:52:18","slug":"using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text","status":"publish","type":"post","link":"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/","title":{"rendered":"Using Perl and Google Chromium’s CLD to identify the language of a text"},"content":{"rendered":"

For a new project I'm working on, given a body of text, I need to identify which language it's written in (English, Russian, Chinese, etc…).<\/p>\n

I'm not exactly the first person on Earth to do this, so it turns out there's Google's CLD library<\/a>. Surprisingly, several people around here didn't know it. The library is open source and very good too, so I immediately looked for Perl bindings for it.<\/p>\n

There is a great Perl module<\/a> on CPAN called Lingua::Identify::CLD<\/a>. This module bundles a copy of the CLD library, and fully automates build and link steps too. So I gave it a shot.<\/p>\n

<\/p>\n

How to use Lingua::Identify::CLD<\/h3>\n

It's amazingly easy to use. Here's a sample of the code:<\/p>\n

\r\n#!\/usr\/bin\/perl\r\n\r\nuse strict;\r\nuse Lingua::Identify::CLD ();\r\n\r\nmy $text;\r\nwhile (readline) { $text .= $_ }\r\nchomp $text;\r\n\r\n# In my case, the content is HTML\r\nmy $cld = Lingua::Identify::CLD->new(isPlainText => 0);\r\n\r\n# Example: (ENGLISH, en, 64)\r\nmy @lang = $cld->identify($text);\r\nsay "Language: $lang[0]";\r\n<\/code><\/pre>\n
<\/p>\n
Failing tests<\/h3>\n
I decided to start using this module into my project. The build phase went fine (perl .\/Build<\/code>), while the tests were failing (.\/Build test<\/code>). Here's the log of a failed test run:<\/p>\n
\r\n$ .\/Build test\r\ncc -I\/usr\/lib\/perl\/5.14\/CORE -fPIC -c -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fstack-protector -fno-strict-aliasing -pipe -I\/usr\/local\/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -o \/tmp\/gAc_glZta2\/library.o \/tmp\/gAc_glZta2\/library.c\r\ncc -I\/usr\/lib\/perl\/5.14\/CORE -fPIC -c -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fstack-protector -fno-strict-aliasing -pipe -I\/usr\/local\/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -o \/tmp\/gAc_glZta2\/test.o \/tmp\/gAc_glZta2\/test.c\r\ncc -shared -L\/usr\/local\/lib -fstack-protector -o \/tmp\/gAc_glZta2\/libfoo.so \/tmp\/gAc_glZta2\/library.o\r\ncc -fstack-protector -L\/usr\/local\/lib -o \/tmp\/gAc_glZta2\/foo \/tmp\/gAc_glZta2\/test.o -L\/tmp\/gAc_glZta2 -lfoo\r\n\r\n** Preparing XS code\r\nt\/00-load.t ....... 1\/1 Bailout called.  Further testing stopped:  \r\n\r\n#   Failed test 'use Lingua::Identify::CLD;'\r\n#   at t\/00-load.t line 6.\r\n#     Tried to use 'Lingua::Identify::CLD'.\r\n#     Error:  Not a CODE reference at \/usr\/lib\/perl\/5.14\/DynaLoader.pm line 207.\r\n# END failed--call queue aborted at ...\/Lingua-Identify-CLD-0.05\/blib\/lib\/Lingua\/Identify\/CLD.pm line 207.\r\n# BEGIN failed--compilation aborted at ...\/Lingua-Identify-CLD-0.05\/blib\/lib\/Lingua\/Identify\/CLD.pm line 24.\r\n# Compilation failed in require at (eval 4) line 2.\r\n# BEGIN failed--compilation aborted at (eval 4) line 2.\r\nUse of uninitialized value $Lingua::Identify::CLD::VERSION in concatenation (.) or string at t\/00-load.t line 9.\r\n# Testing Lingua::Identify::CLD , Perl 5.014002, \/usr\/bin\/perl\r\n# Looks like you failed 1 test of 1.\r\nFAILED--Further testing stopped.\r\n<\/code><\/pre>\n
Just the day before I had successfully compiled and run the tests for the same version of the module, but on Ubuntu 11.10, which I was using. Then I decided to upgrade to 12.10, and that's where I got this failed test run.<\/p>\n
<\/p>\n
Contacting the author<\/h3>\n
Then I decided to contact the author of the module<\/strong>. Being Alberto<\/a> quite a known author, with lots of CPAN contributions, I hoped he would answer my query within 2-3 days. That would give me some time to do other stuff, and hopefully would give him time to analyze the failure.<\/p>\n
As usual with the best CPAN authors ;-) he answered in a couple of hours, which was fantastic for me. He had already identified a few failures like mine thanks to another awesome resource we have in the Perl community, the CPAN Testers service<\/a>.<\/p>\n
<\/p>\n
CPAN Testers<\/h3>\n
CPAN testers is a group of users<\/strong> that regularly (or not) report back the build\/test status of everything that's released to CPAN in a multitude of platforms and versions of Perl. I think this is one of the most underestimated awesome features we have in the Perl community. The CPAN testers status of Lingua::Identify::CLD<\/a> shows one report that looks exactly the same<\/a> as the failure I experienced. This is on Ubuntu 12.10 with the stock perl 5.14.2.<\/p>\n
<\/p>\n
The ugly patch<\/h3>\n
I tried to analyze the problem, apparently located in DynaLoader<\/code>, and came up with a shotgun-debugging-driven<\/i> patch<\/a> that I copy\/paste here for reference:<\/p>\n
@@ -18,10 +18,23 @@ Version 0.05\r\n\r\n our $VERSION = '0.05';\r\n \r\n-use XSLoader;\r\n-BEGIN {\r\n+eval {\r\n+\r\n+    require XSLoader;\r\n     XSLoader::load('Lingua::Identify::CLD', $VERSION);\r\n-}\r\n+\r\n+} or do {\r\n+\r\n+    # This warning triggers on Ubuntu 12.10 with the\r\n+    # stock perl 5.14.2. Strangely enough, this doesn't\r\n+    # seem to affect the tests at all.\r\n+    #\r\n+    # Not a CODE reference at \/usr\/lib\/perl\/5.14\/DynaLoader.pm line 207.\r\n+    # END failed--call queue aborted at ...\/blib\/lib\/Lingua\/Identify\/CLD.pm line 207.\r\n+    # ) at ...\/blib\/lib\/Lingua\/Identify\/CLD.pm line 28."\r\n+    #\r\n+    #warn "Something's wrong with XSLoader? ($@)";\r\n+};\r\n \r\n =head1 SYNOPSIS\r\n<\/code><\/pre>\n
It's shotgun debugging<\/a> because I don't really know what's going on, I just came up with this patch because of the assumptions and information I gathered during the years on how DynaLoader\/XSLoader<\/code> and BEGIN {}<\/code> blocks work or interact with the rest of the code :-)<\/p>\n
Anyway, it makes the tests pass again, even with a weird warning. I agree with Alberto that it's not wise to incorporate this patch into Lingua::Identify::CLD, until we have understood why the original code fails, and why just for 2 people in the world.<\/p>\n
All this blah-blah, to say: please do help! If you have seen the same problem, help us figure out what it is. My repository with the forked\/patched code is on Github:<\/p>\n
https:\/\/github.com\/cosimo\/Lingua-Identify-CLD<\/a><\/p>\n
Have fun!<\/p>\n","protected":false},"excerpt":{"rendered":"
For a new project I'm working on, given a body of text, I need to identify which language it's written in (English, Russian, Chinese, etc…). I'm not exactly the first person on Earth to do this, so it turns out there's Google's CLD library. Surprisingly, several people around here didn't know it. The library is […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[210,49,212,211,209,50],"yoast_head":"\nUsing Perl and Google Chromium's CLD to identify the language of a text - Random hacking<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Using Perl and Google Chromium's CLD to identify the language of a text - Random hacking\" \/>\n<meta property=\"og:description\" content=\"For a new project I'm working on, given a body of text, I need to identify which language it's written in (English, Russian, Chinese, etc…). I'm not exactly the first person on Earth to do this, so it turns out there's Google's CLD library. Surprisingly, several people around here didn't know it. The library is […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/\" \/>\n<meta property=\"og:site_name\" content=\"Random hacking\" \/>\n<meta property=\"article:published_time\" content=\"2012-10-29T06:52:18+00:00\" \/>\n<meta name=\"author\" content=\"cosimo\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"cosimo\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/\"},\"author\":{\"name\":\"cosimo\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"headline\":\"Using Perl and Google Chromium’s CLD to identify the language of a text\",\"datePublished\":\"2012-10-29T06:52:18+00:00\",\"dateModified\":\"2012-10-29T06:52:18+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/\"},\"wordCount\":553,\"commentCount\":1,\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"keywords\":[\"CLD\",\"cpan\",\"cpan-testers\",\"language\",\"Lingua::Identify::CLD\",\"perl\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/\",\"name\":\"Using Perl and Google Chromium's CLD to identify the language of a text - Random hacking\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\"},\"datePublished\":\"2012-10-29T06:52:18+00:00\",\"dateModified\":\"2012-10-29T06:52:18+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Using Perl and Google Chromium’s CLD to identify the language of a text\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\",\"name\":\"Random hacking\",\"description\":\"Assume nothing. Code defensively. Keep it simple, stupid!\",\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\",\"name\":\"cosimo\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"caption\":\"cosimo\"},\"logo\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\"},\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Using Perl and Google Chromium's CLD to identify the language of a text - Random hacking","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/","og_locale":"en_US","og_type":"article","og_title":"Using Perl and Google Chromium's CLD to identify the language of a text - Random hacking","og_description":"For a new project I'm working on, given a body of text, I need to identify which language it's written in (English, Russian, Chinese, etc…). I'm not exactly the first person on Earth to do this, so it turns out there's Google's CLD library. Surprisingly, several people around here didn't know it. The library is […]","og_url":"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/","og_site_name":"Random hacking","article_published_time":"2012-10-29T06:52:18+00:00","author":"cosimo","twitter_card":"summary_large_image","twitter_misc":{"Written by":"cosimo","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/#article","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/"},"author":{"name":"cosimo","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"headline":"Using Perl and Google Chromium’s CLD to identify the language of a text","datePublished":"2012-10-29T06:52:18+00:00","dateModified":"2012-10-29T06:52:18+00:00","mainEntityOfPage":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/"},"wordCount":553,"commentCount":1,"publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"keywords":["CLD","cpan","cpan-testers","language","Lingua::Identify::CLD","perl"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/","url":"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/","name":"Using Perl and Google Chromium's CLD to identify the language of a text - Random hacking","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website"},"datePublished":"2012-10-29T06:52:18+00:00","dateModified":"2012-10-29T06:52:18+00:00","breadcrumb":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2012\/10\/using-perl-and-google-chromiums-cld-to-identify-the-language-of-a-text\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.streppone.it\/cosimo\/blog\/"},{"@type":"ListItem","position":2,"name":"Using Perl and Google Chromium’s CLD to identify the language of a text"}]},{"@type":"WebSite","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website","url":"https:\/\/www.streppone.it\/cosimo\/blog\/","name":"Random hacking","description":"Assume nothing. Code defensively. Keep it simple, stupid!","publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1","name":"cosimo","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","caption":"cosimo"},"logo":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/"},"url":"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/"}]}},"_links":{"self":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/424"}],"collection":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/comments?post=424"}],"version-history":[{"count":0,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/424\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/media?parent=424"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/categories?post=424"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/tags?post=424"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}