Last week, OpenAI released version 2 of an updated neural net called Whisper that approaches human level robustness and accuracy on speech recognition. You can now directly call from R a C/C++ inference engine which allow you to transcribe .wav audio files. To allow to easily do this in R, BNOSAC created an R wrapper around the whisper.cpp code. This R package is available at https //github.com/bnosac/audio.whisper and can be installed as follows.
This week, I uploaded a newer version of the R package recogito to CRAN. The recogito R package provides tools to manipulate and annotate images and text in shiny. It is a htmlwidgets R wrapper around the excellent recogito-js and annotorious javascript libraries as well as it's integration with openseadragon.You can use the package to set up shiny apps which annotate areas of interest (rectangles / polygons) in images with specific labels
Learn how to apply doc2vec in R on your text in this pdf presentation available at https //www.bnosac.be/index.php/blog/103-doc2vec-in-R. Where we focus on our R package doc2vec available at https //github.com/bnosac/doc2vec You can view the presentation below. NEW, since 2020, you can now access courses Text Mining with R and Advanced R programming online through our online school, let us know here if you want to obtain access.
An update of the udpipe R package (https //bnosac.github.io/udpipe/en) landed safely on CRAN last week. Originally the udpipe R package was put on CRAN in 2017 wrapping the UDPipe (v1.2 C++) tokeniser/lemmatiser/parts of speech tagger and dependency parser. It now has many more functionalities next to just providing this parser. The current release (0.8.4-1 on CRAN https //cran.r-project.org/package=udpipe) makes sure default models which are used are the ones trained on version 2.5 of universal dependencies. Other features of the release are detailed in the NEWS item. This is what dependency parsing looks like on some sample text. library(udpipe)x <- udpipe(""The package provides a dependency parsers built on data from universaldependencies.org"", ""english"")View(x)library(ggraph)library(ggplot2)library(igraph)library(textplot)plt <- textplot_dependencyparser(x, size = 4, title = ""udpipe R package - dependency parsing"")plt
Finally, the R package you all have been waiting for has arrived - image.ContourDetector developed at https //github.com/bnosac/image. It detects contour lines in images alongside the 'Unsupervised Smooth Contour Detection' algorithm available at http //www.ipol.im/pub/art/2016/175. Have you always wanted to be able to draw like you are in art school? Let me show how to quickly do this.
Learn how to apply word2vec in R on your text in this pdf presentation available at https //www.bnosac.be/index.php/blog/100-word2vec-in-R. Where we focus on our R package word2vec available at https //github.com/bnosac/word2vec You can view the presentation below. NEW, since 2020, you can now access courses Text Mining with R and Advanced R programming online through our online school, let us know here if you want to obtain access.
A few weeks ago, we pushed R package textplot to CRAN and it was accepted for release last week. The package contains straightforward functionalities for the visualisation of text, namely of text cooccurences text clusters (in casu biterm clusters)
A few weeks ago, we published an update of the BTM (Biterm Topic Models for text) package on CRAN. Biterm Topic Models are especially usefull if you want to find topics in collections of short texts. Short texts are typically a twitter message, a short answer on a survey, the title of an email, search questions, ... . For these types of short texts traditional topic models like Latent Dirichlet Allocation are less suited as most information is available in short word combinations. The R package BTM finds topics in such short texts by explicitely modelling word-word co-occurrences (biterms) in a short window. The update which was pushed to CRAN a few weeks ago now allows to explicitely provide a set of biterms to cluster upon. Let us show an example on clustering a subset of R package descriptions on CRAN. The resulting cluster visualisation looks like this.
I lost a few hours this afternoon when digging into the Corona virus data mainly caused by reading this article at this website which gives a nice view on how to be aware of potential issues which can arise when collecting data and to be aware of hidden factors and it also shows Belgium. As a Belgian, I was interested to see how Corona might impact our lives in the next weeks and out of curiosity I was interested to see how we are doing compared to other countries regarding containment of the Corona virus outspread - especially since we still do not have a government in Belgium after elections 1 year ago. In what follows, I'll be showing some graphs using data available at https //github.com/CSSEGISandData/COVID-19 (it provides up-to-date statistics on Corona cases). If you want to reproduce this, pull the repository and just execute the following R code shown.
Next week we will hold our yearly course on Advanced R programming at LStat, Leuven. If you are interested in learning one of the following techniques, don't hesitate to subscribe at https //lstat.kuleuven.be/training/coursedescriptions/AdvancedprogramminginR.html Functions, the apply family of functions, parallelisation, advanced data manipulation with R S3 programming
Last week, we updated package ruimtehol on CRAN. The package provides an easy interface for R users to Starspace which is a general purpose neural embedding model for text data. Notable changes are that the package now also builds fine on Mac OS and runs fine on all CRAN platforms. If you are interested to see what the package can do, have a look at the presentation below or visit the package vignette at https //cran.r-project.org/web/packages/ruimtehol/vignettes/ground-control-to-ruimtehol.pdf
I forgot to do some marketing for the following upcoming AI-related courses which will be given in Leuven, Belgium by BNOSAC 2019-10-17&18 Statistical Machine Learning with R Subscribe here 2019-11-14&15 Text Mining with R Subscribe here
We have been blogging about udpipe several times now in the following posts update of udpipe You did a sentiment analysis with tidytext but you forgot to do dependency parsing to answer WHY is something positive/negative
Last week the R package ruimtehol was updated on CRAN giving R users who perform Natural Language Processing access to the possibility to Allow to do semi-supervised learning (learning where you have both text as labels but not always both of them on the same document identifier. Allow to do transfer learning by passing on an embedding matrix (e.g. obtained via fasttext or Glove or the like) and keep on training based on that matrix or just use the embeddings in your Natural Language Processing flow.
Last call for the course on Text Mining with R, held next week in Leuven, Belgium on April 1-2. Viewing the course description as well as subscription can be done at https //lstat.kuleuven.be/training/coursedescriptions/text-mining-with-r Some things you'll learn ... is that King Filip of Belgium is similar to public expenses if we just look at open data from questions and answers in Belgian parliament (retrieved from here http //data.dekamer.be). Proof is below. See you next week. library(ruimtehol)library(data.table)library(lattice)library(latticeExtra)data(""dekamer"", package = ""ruimtehol"")dekamer$x <- strsplit(dekamer$question, ""\\W"")dekamer$x <- lapply(dekamer$x, FUN = function(x) setdiff(x, """"))dekamer$x <- sapply(dekamer$x, FUN = function(x) paste(x, collapse = "" ""))dekamer$x <- tolower(dekamer$x)dekamer$y <- strsplit(dekamer$question_theme, split = "","")dekamer$y <- lapply(dekamer$y, FUN=function(x) gsub("" "", ""-"", x))set.seed(321)model <- embed_tagspace(x = dekamer$x, y = dekamer$y, early_stopping = 0.8, validationPatience = 10, dim = 50, lr = 0.01, epoch = 40, loss = ""softmax"", adagrad = TRUE, similarity = ""cosine"", negSearchLimit = 50, ngrams = 2, minCount = 2)embedding_words <- as.matrix(model, type = ""words"")embedding_labels <- as.matrix(model, type = ""labels"", prefix = FALSE)embedding_person <- starspace_embedding(model, tolower(c(""Theo Francken"")))embedding_person <- starspace_embedding(model, tolower(c(""Koning Filip"")))similarities <- embedding_similarity(embedding_person, embedding_words, top = 9)similarities <- subset(similarities, !term2 %in% c(""koning"", ""filip""))similarities$term <- factor(similarities$term2, levels = rev(similarities$term2))plt1 <- barchart(term ~ similarity | term1, data = similarities, scales = list(x = list(relation = ""free""), y = list(relation = ""free"")), col = ""darkgreen"", xlab = ""Similarity"", main = ""Koning Filip lijkt op ..."")similarities <- embedding_similarity(embedding_person, embedding_labels, top = 7)similarities$term <- factor(similarities$term2, levels = rev(similarities$term2))plt2 <- barchart(term ~ similarity | term1, data = similarities, scales = list(x = list(relation = ""free""), y = list(relation = ""free"")), col = ""darkgreen"", xlab = ""Similarity"", main = ""Koning Filip lijkt op ..."")c(plt1, plt2)
Doing human face detection with computer vision is probably something you do once unless you work for police departments, you work in the surveillance industry or for the Chinese government. In order to reduce the time you lose on that small exercise, bnosac created a small R package (source code available at https //github.com/bnosac/image) which wraps the weights of a Single Shot Detector (SSD) Convolutional Neural Network which was trained with the Caffe Deep Learning kit. That network allows to detect human faces in images. An example is shown below (tested on Windows and Linux). install.packages(""magick"")install.packages(""image.libfacedetection"", repos = ""https //bnosac.github.io/drat"") library(magick)library(image.libfacedetection)image <- image_read(""http //bnosac.be/images/bnosac/blog/wikipedia-25930827182-kerry-michel.jpg"")faces <- image_detect_faces(image)facesplot(faces, image, border = ""red"", lwd = 7, col = ""white"")
For people from Belgium working in R with spatial data, you can find excellent workshop material on creating thematic maps for Belgium at https //workshop.mhermans.net/thematic-maps-r/index.html. The workshop was given by Maarten Hermans from HIVA - Onderzoeksinstituut voor Arbeid en Samenleving. The plots are heavily based on BelgiumMaps.Statbel - an R package from bnosac released 2 years ago (more info at http //www.bnosac.be/index.php/blog/55-belgiummaps-statbel-r-package-with-administrative-boundaries-of-belgium)
At BNOSAC, R is used a lot to perform text analytics as it is an excellent tool that provides anything a data scientist needs to perform data analysis on text in a business settings. For users unfamiliar with all the possibilities that the wealth of R packages offers regarding text analytics, we've made this small mindmap showing a list of techniques and R packages that are used frequently in text mining projects set up by BNOSAC. Download the image and let your eyes zoom in on the different topics. Hope it broadens your idea of what is possible. Want to learn more or get hands on http //www.bnosac.be/index.php/training
Last week the R package ruimtehol was released on CRAN (https //github.com/bnosac/ruimtehol) allowing R users to easily build and apply neural embedding models on text data. It wraps the 'StarSpace' library https //github.com/facebookresearch/StarSpace allowing users to calculate word, sentence, article, document, webpage, link and entity 'embeddings'. By using the 'embeddings', you can perform text based multi-label classification, find similarities between texts and categories, do collaborative-filtering based recommendation as well as content-based recommendation, find out relations between entities, calculate graph 'embeddings' as well as perform semi-supervised learning and multi-task learning on plain text. The techniques are explained in detail in the paper 'StarSpace Embed All The Things!' by Wu et al. (2017), available at https //arxiv.org/abs/1709.03856. You can get started with some common text analytical use cases by using the presentation we have built below. Enjoy!
A small note on the growing list of users of the udpipe R package. In the last month of 2018, we've updated the package on CRAN with some noticeable changes The default models which are now downloaded with the function udpipe_download_model are now models built on Universal Dependencies 2.3 (released on 2018-11-15) This means udpipe now has models for 60 languages. That's right! And they provide tokenisation, parts of speech tagging, lemmatisation and dependency parsing built on all of these treebanks afrikaans-afribooms, ancient_greek-perseus, ancient_greek-proiel, arabic-padt, armenian-armtdp, basque-bdt, belarusian-hse, bulgarian-btb, buryat-bdt, catalan-ancora, chinese-gsd, coptic-scriptorium, croatian-set, czech-cac, czech-cltt, czech-fictree, czech-pdt, danish-ddt, dutch-alpino, dutch-lassysmall, english-ewt, english-gum, english-lines, english-partut, estonian-edt, finnish-ftb, finnish-tdt, french-gsd, french-partut, french-sequoia, french-spoken, galician-ctg, galician-treegal, german-gsd, gothic-proiel, greek-gdt, hebrew-htb, hindi-hdtb, hungarian-szeged, indonesian-gsd, irish-idt, italian-isdt, italian-partut, italian-postwita, japanese-gsd, kazakh-ktb, korean-gsd, korean-kaist, kurmanji-mg, latin-ittb, latin-perseus, latin-proiel, latvian-lvtb, lithuanian-hse, maltese-mudt, marathi-ufal, north_sami-giella, norwegian-bokmaal, norwegian-nynorsk, norwegian-nynorsklia, old_church_slavonic-proiel, old_french-srcmf, persian-seraji, polish-lfg, polish-sz, portuguese-bosque, portuguese-br, portuguese-gsd, romanian-nonstandard, romanian-rrt, russian-gsd, russian-syntagrus, russian-taiga, sanskrit-ufal, serbian-set, slovak-snk, slovenian-ssj, slovenian-sst, spanish-ancora, spanish-gsd, swedish-lines, swedish-talbanken, tamil-ttb, telugu-mtg, turkish-imst, ukrainian-iu, upper_sorbian-ufal, urdu-udtb, uyghur-udt, vietnamese-vtb.
Our recent addition to the NLP R universe is called R package ruimtehol which is open sourced at https //github.com/bnosac/ruimtehol This R package is a wrapper around Starspace which provides a neural embedding model for doing the following on text Text classification Learning word, sentence or document level embeddings
A new R package called crfsuite supported by BNOSAC landed safely on CRAN last week. The crfsuite package (https //github.com/bnosac/crfsuite) is an R package specific to Natural Language Processing and allows you to easily build and apply models for named entity recognition text chunking
Last call for the 2-day course on Text Mining with R, held next week (08-09 October 2018) in Brussels, Belgium. Subscribe at https //www.eventbrite.co.uk/e/dsb2018-text-mining-with-r-jan-wijffels-bnosac-session-03-04-tickets-50586501588 You'll learn during that course the following Cleaning of text data, regular expressions
This blogpost announces the release of the udpipe R package version 0.7 on CRAN. udpipe is an R package which does tokenization, parts of speech tagging, lemmatization, morphological feature tagging and dependency parsing. It's main feature is that it is a lightweight R package which works on more than 50 languages and gives you rich NLP output out of the box. The package was updated mainly in order to more easily work with the crfsuite R package which does entity/intent recogntion and chunking. The user-visible changes that were made are that udpipe now has a shorthand for working with text in the TIF format and it now also allows to indicate the location of the token inside the original text. Next to this, version 0.7 also caches the udpipe models. Example
Yesterday there was a pretty controversial Pano TV documentary called 'Wie is Schild & Vrienden echt' at the national television channel 'één' (https //www.vrt.be/vrtnu/a-z/pano/2018/pano-s2018a10). The documentary revealed the internal communication of a right-wing group from Belgium, called
I'm happy to announce that the following list of courses for R users is ready to be booked. All courses are face-to-face courses held in Belgium. 08-09/10/2018 Text mining with R. Brussels (Belgium). http //di-academy.com/bootcamp + send mail to training@di-academy.com 15-16/10/2018 Statistical machine learning with R. Leuven (Belgium). Subscribe here
Last Wednesday, a small presentation was given at the RBelgium meetup in Brussels on Basic R Automation. For those of you who could not attend, here are the slides of that presentation which showed the use of the cronR and taskscheduleR R packages for automating basic R scripts. If you are interested in setting up a project for more advanced ways on how to automate your R processes for your specific environment, get in touch. {aridoc engine=""pdfjs"" width=""100%"" height=""550""}images/bnosac/blog/Basic_R_Automation.pdf{/aridoc}
In this blogpost, we will show 6 keyword extraction techniques which allow to find keywords in plain text. Keywords are frequently occuring words which occur somehow together in plain text. Common examples are New York, Monte Carlo, Mixed Models, Brussels Hoofdstedelijk Gewest, Public Transport, Central Station, p-values, ... If you master these techniques, it will allow you to easily step away from doing simple word frequency statistics to more business-relevant text summarisation. For this, we will use the udpipe R package (docs at https //CRAN.R-project.org/package=udpipe or https //bnosac.github.io/udpipe/en) which is the core R package you need for doing this type of t ext processing.We'll basically show how to easily extract keywords as follows 1. Find keywords by doing Parts of Speech tagging in order to identify nouns2. Find keywords based on Collocations and Co-occurrences3. Find keywords based on the Textrank algorithm4. Find keywords based on RAKE (rapid automatic keyword extraction)5. Find keywords by looking for Phrases (noun phrases / verb phrases)6. Find keywords based on results of dependency parsing (getting the subject of the text)
Last week we updated the cronR R package and released it to CRAN allowing you to schedule any R code on whichever timepoint you like. The package was updated in order to comply to more stricter CRAN policies regarding writing to folders. Along the lines, the RStudio add-in of the package was also updated. It now looks as shown below and is tailored to Data Scientists that want to automate basic R scripts.
In the last few years, Natural Language Processing (NLP) has become more and more an open multi-lingual task instead of being held back by language, country and legal boundaries. With the advent of commonly used open data regarding natural language processing tasks as available at http //universaldependencies.org one can now relatively easily compare different toolkits which perform natural language processing. In this post we compare the udpipe R package to the spacyr R package. UDPipe - spaCy comparison A traditional natural language processing flow consists of a number of building blocks which can be used to structure your Natural Language Application on top of it. Namely
Last call for the course on Advanced R programming scheduled in Leuven, Belgium on Febuary 20-21 2018. Register at https //lstat.kuleuven.be/training/coursedescriptions/AdvancedprogramminginR.html You'll learn during that course
If you work on applications for clients or have open sourced some shiny apps, a question that arises is how is your application being used. What you can do in order to find out how your hard work is being consumed is putting your code in logs and then viewing the logs. An easier way however to track usage of your application is just sending page views or application events to Google Analytics. That's exactly what the GAlogger R package (https //github.com/bnosac/GAlogger) is doing. It allows to log R events and R usage to Google Analytics and was created with the following use cases in mind Track usage of your application
BNOSAC is happy to announce the release of the udpipe R package (https //bnosac.github.io/udpipe/en) which is a Natural Language Processing toolkit that provides language-agnostic 'tokenization', 'parts of speech tagging', 'lemmatization', 'morphological feature tagging' and 'dependency parsing' of raw text. Next to text parsing, the package also allows you to train annotation models based on data of 'treebanks' in 'CoNLL-U' format as provided at http //universaldependencies.org/format.html. Language models The package provides direct access to language models trained on more than 50 languages. The following languages are directly available
BNOSAC is working on building an application on top of open data from questions and answers given at the parliament in Belgium. It will basically show what our civil servants in parliament are busy with. If you are interested in co-developing, feel free to get in touch for a quick chat. For those of you interested in an overview of open data available in Belgium, we've made a presentation showing what open data is available in Belgium for direct use (see below). Interested in how open data can be used for your business, get in touch. {aridoc engine=""pdfjs"" width=""100%"" height=""550""}images/bnosac/blog/open_data_be.pdf{/aridoc}
CRAN contains up to date (October 2017) more than 11500 R packages. If you want to scroll through all of these, you probably need to spend a few days, assuming you need 5 seconds per package and there are 8 hours in a day. Since R version 3.4, we can also get a dataset will all packages, their dependencies, the package title, the description and even the installation errors which the packages have. Which makes the CRAN database with all packages an excellent dataset for doing text mining. If you want to get that dataset, just do as follows in R library(tools)crandb <- CRAN_package_db()
We use text mining a lot in day-to-day data mining operations. In order to share our knowledge on this, to show that R is an extremely mature platform to do business-oriented text analytics and to give you practical experience with text mining, our course on Text Mining with R is scheduled for the 3rd consecutive year at LStat, the Leuven Statistics Research Center (Belgium) as well as at the Data Science Academy in Brussels. Courses are scheduled 2 times in November 2017 and also in March 2018. This course is a hands-on course covering the use of text mining tools for the purpose of data analysis. It covers basic text handling, natural language engineering and statistical modelling on top of textual data. The following items are covered. Text encodings
If you work on natural language processing in a day-to-day setting which involves statistical engineering, at a certain timepoint you need to process your text with a number of text mining procedures of which the following ones are steps you must do before you can get usefull information about your text Tokenisation (splitting your full text in words/terms) Parts of Speech (POS) tagging (assigning each word a syntactical tag like is the word a verb/noun/adverb/number/...)
For R users interested in Machine Learning, you can attend our upcoming course on Machine Learning with R which is scheduled on 18-19 October 2017 in Leuven, Belgium. This is now the 4th year this course is given at the university of Leuven so we made quite some updates since the first time this was given 4 years ago. During the course you'll learn the following techniques from a methodological as well as a practical perspective naive bayes, trees, feed-forward neural networks, penalised regression, bagging, random forests, boosting and if time permits graphical lasso, penalised generalised additive models, support vector machines. Subscribe here https //lstat.kuleuven.be/training/coursedescriptions/statistical-machine-learning-with-r
Just before the summer holidays, BNOSAC presented a talk called Computer Vision and Image Recognition algorithms for R users at the UseR conference. In the talk 6 packages on Computer Vision with R were introduced in front of an audience of about 250 persons. The R packages we covered and that were developed by BNOSAC are image.CornerDetectionF9 FAST-9 corner detection image.CannyEdges Canny Edge Detector
Parts of Speech (POS) tagging is a crucial part in natural language processing. It consists of labelling each word in a text document with a certain category like noun, verb, adverb, pronoun, ... . At BNOSAC, we use it on a dayly basis in order to select only nouns before we do topic detection or in specific NLP flows. For R users working with different languages, the number of POS tagging options is small and all have up or downsides. The following taggers are commonly used. The Stanford Part-Of-Speech Tagger which is terribly slow, the language set is limited to English/French/German/Spanish/Arabic/Chinese (no Dutch). R packages for this are available at http //datacube.wu.ac.at. Treetagger (http //www.cis.uni-muenchen.de/~schmid/tools/TreeTagger) contains more languages but is only usable for non-commercial purposes (can be used based on the koRpus R package)
2 new R packages were put on CRAN last week by BNOSAC (www.bnosac.be). One package for scheduling R scripts and processes on Windows (taskscheduleR) and Another package for scheduling R scripts and processes on Unix / Linux (cronR)
As part of our data science training initiative, bnosac is also providing a course on computer vision with R & Python which is held in March 9-10 in Leuven, Belgium (subscribe here or have a look at our full training offer here). Part of the course is covering finding blobs, corners, gradients, edges & lines in images. For this reason, the R package image.LineSegmentDetector was made available at https //github.com/bnosac/image. It allows to detect segment lines in digital images. An example of this is shown below. library(image.LineSegmentDetector)library(pixmap)## Read in the image + make sure input to the algorithm is matrix with grey-scale values in 0-255 rangeimagelocation <- system.file(""extdata"", ""chairs.pgm"", package=""image.LineSegmentDetector"")image <- read.pnm(file = imagelocation, cellres = 1)x <- image@grey * 255## Detect and plot the lines segmentslinesegments <- image_line_segment_detector(x)linesegmentsplot(image)plot(linesegments, add = TRUE, col = ""red"")
Tomorrow, the next RBelgium meeting will be held at the bnosac offices. This is the schedule. Interested? Feel free to join the event. More info http //www.meetup.com/RBelgium/events/228427510/ • 18h00-18h30 enter & meet other R users
If you are working on a Windows computer and want to schedule your R scripts while you are off running, sleeping or having a coffee break, the taskscheduleR package might be what you are looking for. The taskscheduleR R package is available at https //github.com/bnosac/taskscheduleR and it allows R users to do the following
With the release of RStudio add-in possibilities, a new area of productivity increase and expected new features for R users has arrived. Thanks to the help of Oliver who has written an RStudio add-in on top of taskscheduleR, scheduling and automating an R script from RStudio is now exactly one click away if you are working on Windows. How? Just install these R packages and you have the add-in ready at the add-in tab in your RStudio session. Select your R script and schedule it to run any time you want. Hope this saves you some day-to-day time and feel free to help make additional improvements. More information https //github.com/bnosac/taskscheduleR.
Within 2 weeks, our 2-day crash course on Applied spatial modelling with R (April 13-14, 2016) will be given at the University of Leuven, Belgium https //lstat.kuleuven.be/training/applied-spatial-modelling-with-rYou'll learn during this course the following elements The sp package to handle spatial data (spatial points, lines, polygons, spatial data frames) Importing spatial data and setting the spatial projection
As part of our continuing effort to digitise poetry and to automate new forms of poetry, we released an R package called pattern.nlp, which is available at https //github.com/bnosac/pattern.nlp . It allows R users to do sentiment analysis and Parts of Speech tagging for text written in Dutch, French, English, German, Spanish or Italian. Of course this can also be used for other purposes like data preparation as part of a topic modelling flow. If you are interested in text mining, feel free to register for the text mining courses listed at our last blog post.
We recently opened up the BelgiumMaps.StatBel package and made it available at https //github.com/bnosac/BelgiumMaps.StatBel. This R package contains maps with administrative boundaries (national, regions, provinces, districts, municipalities, statistical sectors, agglomerations (200m)) of Belgium extracted from Open Data at Statistics Belgium. The package is a data-only package where maps of administrative zones in Belgium are available in the WGS84 coordinate reference system. The data is available in several objects
Text Mining has become quite mainstream nowadays as the tools to make a reasonable text analysis are ready to be exploited and give astoundingly nice and reasonable results. At BNOSAC, we use it mainly for text mining on call center data, poetry, salesforce data, emails, HR reviews, IT logged tickets, customer reviews, survey feedback and many more. We also provide training on text mining with R in Belgium (next trainings are scheduled on November 14/15 2016 and March 23/24 2017 in Belgium, to subscribe go here or here). For this blog post, we will focus on the most relevant visualisations which exist in text mining by using the CETA trade agreement between the EU and Canada.
Part of the R course offering of BNOSAC which you can find at http //bnosac.be/images/bnosac/bnosac_courses_r.pdf, we offer several 2-day hands-on courses covering the use of text mining tools for the purpose of data analysis. It covers basic text handling, natural language engineering and statistical modelling on top of textual data. Interested in upgrading your skills on text mining with R? Registering can be done for the following days. 2016 October 24-25 subscribe at https //lstat.kuleuven.be/training/coursedescriptions/text-mining-with-r2016 November 14-15 subscribe at http //di-academy.com/event/text-mining-with-r/2017 March 23-24 subscribe at https //lstat.kuleuven.be/training/coursedescriptions/text-mining-with-r
In a recent post by Revolution Analytics (link & link) in which Revolution was benchmarking their closed source generalized linear model approach with SAS, Hadoop and open source R, they seemed to be pointing out that there is no 'easy' R open source solution which exists for building a poisson regression model on large datasets. This post is about showing that fitting a generalized linear model to large data in R <is> easy in open source R and just works.
A few weeks ago, the stream package has been released on CRAN. It allows to do real time analytics on data streams. This can be very usefull if you are working with large datasets which are already hard to put in RAM completely, let alone to build some statistical model on it without getting into RAM problems. Most of the standard statistical algorithms require access to all data points and make several iterations over the data and are less suited for usage in R on big datasets.
A few weeks ago, Rstudio released it's download logs, showing who downloaded R packages through their CRAN mirror. More info http //blog.rstudio.org/2013/06/10/rstudio-cran-mirror/ This is very nice information and it can be used to show the popularity of packages with R, which has been done before and criticized also as the RStudio logs might/might not be representative for the download behaviour of all useRs.
Myrrix is probably more known by java developers and users of Mahout than R users. This is because most of the times java and R developers live in a different community. If you go to the website of Myrrix (http //myrrix.com), you'll find out that it is a large-scale recommender system which is able to build a recommendation model based on Alternating Least Squares. That technique is a pretty good benchmark model if you tune it well enough to get recommendations to your customers.
Advanced R programming topics Similarly as last year, BNOSAC is offering the short course on 'Advanced R programming topics' at the Leuven Statistics Research Center (Belgium). The course is now part of FLAMES (Flanders Training Network for Methodology and Statistics) and can be found here http //www.flames-statistics.eu/training/advanced-r-programming-topics. Subscription is no longer possible unless you ask kindly to LStat.
Within 2 weeks on Thursday, March 20, The RBelgium R user group is holding its next Regular meeting in Brussels for which this is the schedule ** Analysis and visualisation of climate data from the atmospheric model ALADIN using the Rfa package! (Rozemien De Troch - Onderzoeksdepartement KMI) ** Probabilistic latent feature analysis with the plfm package (Michel Meulders - Centre for Information Management, Modeling and Simulation, KU Leuven@ HUBrussel)
For those of you who don't know MOA. MOA stands for Massive On-line Analysis and is an open-source framework that allows to build and run experiments of machine learning or data mining on evolving data streams. The website of MOA (http //moa.cms.waikato.ac.nz) indicates it contains machine learning algorithms for classification, regression, clustering, outlier detection and recommendation engines. For R users who work with a lot of data or encounter RAM issues when building models on large datasets, MOA and in general data streams have some nice features. Namely
Last week, we released the RMOA package at CRAN (http //cran.r-project.org/web/packages/RMOA). It is an R package to allow building streaming classification and regression models on top of MOA. MOA is the acronym of 'Massive Online Analysis' and it is the most popular open source framework for data stream mining which is being developed at the University of Waikato http //moa.cms.waikato.ac.nz. Our RMOA package interfaces with MOA version 2014.04 and focusses on building, evaluating and scoring streaming classification & regression models on data streams. Classification & regression models which are possible through RMOA are - Classification trees * AdaHoeffdingOptionTree * ASHoeffdingTree * DecisionStump * HoeffdingAdaptiveTree * HoeffdingOptionTree * HoeffdingTree * LimAttHoeffdingTree * RandomHoeffdingTree - Bayesian classification * NaiveBayes * NaiveBayesMultinomial - Active learning classification * ActiveClassifier - Ensemble (meta) classifiers * Bagging + LeveragingBag + OzaBag + OzaBagAdwin + OzaBagASHT * Boosting + OCBoost + OzaBoost + OzaBoostAdwin * Stacking + LimAttClassifier * Other + AccuracyUpdatedEnsemble + AccuracyWeightedEnsemble + ADACC + DACC + OnlineAccuracyUpdatedEnsemble + TemporallyAugmentedClassifier + WeightedMajorityAlgorithm - Regression modelling * AMRulesRegressor * FadingTargetMean * FIMTDD * ORTO * Perceptron * RandomRules * SGD (Stochastic Gradient Descent) * TargetMean Interfaces are implemented to model data in standard files (csv, txt, delimited), ffdf data (from the ff package), data.frames and matrices. Documentation of MOA directed towards RMOA users can be found at http //jwijffels.github.io/RMOA Examples on the use of RMOA can be found in the documentation, on github at https //github.com/jwijffels/RMOA or e.g. by viewing the showcase at http //bnosac.be/index.php/blog/16-rmoa-massive-online-data-stream-classifications-with-r-a-moa If you need support on building streaming models on top of your large dataset. Get into contact.
This year, BNOSAC offers 2 R courses in cooperation with the Leuven Statistics Research Center. The courses are part of the Leuven STATistics STATe of the Art Training Initiative and are given in Leuven (Belgium). For R users and data scientists we offer a 2 short courses on R programming & statistical learning. Namely
CRAN mirrors are the backbone to everyday common R usage. They provide the R website and most of the R packages today. Currently there are about 104 official CRAN mirrors. Hosting a CRAN mirror is one step to help the R community and is explained here. To ease that process, at BNOSAC, we have created a Docker image which sets up a CRAN mirror. That Docker image can be found and is available for download at the following docker registry https //registry.hub.docker.com/u/bnosac/cran-mirror For people who don't know Docker, it is basically a tool which allows developers to containerise an application. In this case, the application is to run a CRAN mirror. How does it work. 3 steps 1. Install Docker on your computer or server as explained here, if you haven't done this already. 2. Pull the docker image docker pull bnosac/cran-mirror 3. Run the CRAN mirror docker run -p 22 22 -p 80 80 -v /home/bnosac/CRAN/ /var/www/html -d bnosac/cran-mirror
The next RBelgium meetup will be about R & Google Maps and R & Robotics (ROS). BNOSAC will be hosting the event this time. This is the schedule • 17h30-18h open questions • 18h-19h R and Google Maps • 19h-20h R and Robotics (ROS) For practical information http //www.meetup.com/RBelgium/events/220918123
For those of you who are interested in using R alongside Robotics applications and want to use ROS (Robot Operating System) together with R. Enjoy the slides of our presentation on this topic during the last RBelgium meetup.
For those of you who are interested in using R alongside Google Maps by using the packages geonames (www.geonames.org), RgoogleMaps, ggmap, loa and plotKML. Enjoy the slides of our presentation on this topic during the last RBelgium meetup.
We will be showcasing our RMOA package at the next R User conference in Aalborg. For the R users who are unfamiliar with streaming modelling and want to be ahead of the Gartner Hype cycle or want to evaluate existing streaming machine learning models, RMOA allows to build, run and evaluate streaming classification models which are built in MOA (Massive Online Learning). For an introduction to RMOA and MOA and the type of machine learning models which are possible in MOA - see our previous blog post or scroll through our blog page. In this example below, we showcase the RMOA package by using streaming JSON data which can come from whatever noSQL database that spits out json. For this example, package jsonlite provides a nice stream_in function (an example is shown here) which handles streaming json data. Plugging in streaming machine learning models with RMOA is a breeze. Let's dive into the R code immediately where we show how to run, build and evaluate a streaming boosted classification model.
BNOSAC will be giving from June 08 up to June 12 a 5-day crash course on the use of R using Oracle R Enterprise. The course is given together with our Oracle Partner in Leuven, Belgium. If you are interested in attending, contact us for further details. For R users who aren't aware of this yet. Oracle has embedded R into it's database which allows R users to transparently run R code inside the database - yes really transparently. The Oracle R Enterprise is part of the Oracle Advanced Analytics stack which basically consists of the following elements for R users
For persons interested in advancing their knowledge on R and data science with R, BNOSAC offers a range of courses for R users. These cover courses on R analytics (Statistical Machine Learning with R, Text mining with R, Applied Spatial modelling with R) R programming (R for starters, Common data manipulation for R programmers, Reporting with R, Creating R packages and R repositories, Managing R processes, Using SVN/git with RStudio, Data connectivity using R, Integration of R into web applications)
Waw, our course list for teaching R is getting bigger and bigger. We have now courses on basic, R, advanced R, R package building, statistical machine learning with R, text mining with R and spatial analysis with R. All face-to-face courses given in Belgium and scheduled in the coming months. Some courses are given at the European Data Innovation Hub (Brussels, Belgium), other courses are given through the Leuven Statistics Research Center (Leuven, Belgium). From today on, you can register for the following courses regarding the use of R. Prices are set to 300€ per course day + taxes. For detailed information on the course content, have a look at the pdf which can be found here. Courses given at the European Data Innovation Hub (Brussels, Belgium) - http //www.datainnovationhub.eu
Last week, we had a great course on Text Mining with R at the European Data Innovation Hub. For persons interested in text mining with R, another 1-day crash course is scheduled at the Leuven Statistics Research Center (Belgium) on November 17 (http //lstat.kuleuven.be/training/coursedescriptions/text-mining-with-r). The following elements are covered in the course. Import of (structured) text data with focus on text encodings. Detection of language Cleaning of text data, regular expressions
On 22/10/2015, the Belgium government launched its Open Data initiative by releasing a number of datasets related to population statistics, fiscal information, 'kadaster', the 2011 census and some tools. Because BNOSAC works a lot with these kind of data and because we like to promote open data, an R package called BelgiumStatistics was made available for R users at https //github.com/jwijffels/BelgiumStatistics The package contains all the datasets released by Statistics Belgium (Bevolking, Werk, Leefmilieu, Census 2011) under the 'Licentie open data'. Readily available to R users. Thanks to the open data, analysing and visualising Belgium data has now become a lot smoother as the example below shows.
Last call for registration of the course on Advanced R programming topics. Next week on February 17/18, the yearly R course on Advanced R programming topics in Leuven (Belgium) is scheduled. Registration can be done at https //lstat.kuleuven.be/training/coursedescriptions/AdvancedprogramminginR.html. The course is rewritten based on 3 years of extensive customer feedback and because of the tremendous evolution R has encountered in the last years. You'll learn the following in this 2-day course functions and vectorisation control flow
The R package ETLUtils is now available for download at it's CRAN repository.It's a package which facilitates the ETL in situations where you need to interact with SQL databases in a corporate environment. Basically it currently focusses on the E(Extract) part of the ETL. In the libary you'll find a function called read.dbi.ffdf which allows to extract easily large datasets from an SQL database through DBI straight into a ffdf (part of the ff package). This should ease the usage of the ff package for working with large datasets in corporate environments where you need to interact e.g. with MySQL, PostgreSQL, Oracle through the package RMySQL, RPostgreSQL or ROracle without having to worry about RAM issues when importing large datasets.
The ff package is great when you are working with large data in R. Data in corporate environments are usually not that large that a Hadoop system is needed to handle it but the data are mostly large enough to make R choke on it's RAM. The ff package is great for this type of data. It can handle 2.14 billion elements per atomic (so 2.14 billion records). This is mostly more than enough in standard business settings nowadays. For a while getting the data in R was quite cumbersome. The ff package now provides already nice interfaces to csv and flat files in the read.csv.ffdf and read.table.ffdf functions. To extend this towards standard SQL databases, the ETLUtils package has been put at CRAN. It contains a function called read.dbi.ffdf which can be used to extract easily data from any sql database through DBI directly into an ffdf in R.
If you are into large data but not enormeoulsy big data everyone is talking about and you are tired of finding a solution to get your data with several 10's of millions of records in R without having RAM issues, having a look at the packages ff, ffbase and ETLUtils might be the solution to your problem. Following up on our post about the ETLUtils package which eases transferring large data from SQL databases to ffdf objects in R, the ETLUtils package has now been extended to include the function read.odbc.ffdf which can be used to fetch your SQL queries on corporate Oracle, MySQL, PostgreSQL & sqlite databases. Below we show an example where read.dbi.ffdf is used to fetch all rows of a table and we add data of the same structure with read.odbc.ffdf to the existing ffdf. This might be of interest to you if you work a lot with dayly incremental data updates.
The ff package is a great and efficient way of working with large datasets. One of the main reasons why I prefer to use it above other packages that allow working with large datasets is that it is a complete set of tools. When comparing it to the other open source 'bigdata' packages in R
Every year, the Leuven Statistics Research Center (Belgium) is offering short courses for professionals and researchers in statistics and statistical tools. The following link shows the overview of the courses http //lstat.kuleuven.be/consulting/shortcourses/ENcourse%20overview.htm or get it here in pdf http //lstat.kuleuven.be/consulting/shortcourses/BRO_LSTAT_2012-2013.pdf
Next week on Friday, November 16, the RBelgium R user group is holding its next Regular meeting in Brussels. This is the schedule of the upcoming RBelgium Regular meeting * Graphical User Interface developments around R, including tcltk2 and SciViews - Philippe Grosjean (UMons)* Using R via the Amazon Cloud - Jean-Baptiste Poullet (stat'Rgy)* Literature review R books - Brecht Devleesschauwer (UGent, UCL)The meeting will take place on Friday 16 November, at 18h45, at the ULB Campus de la Plaine. Everyone is welcome to join!
