Me: So I can go with any mobile technology I want, correct?
Him: Yes, most people do Android projects, but I've seen a few successful ones for iOS.
Me: I have a Nokia 3310 refurbished and it should be possible to write apps for that, _somehow_.
It took me a couple evenings to set things up correctly and once I got a reliable environment going, everything got much easier. (Or at least as easy as things can be when developing things for a 10-year-old platform with a heavily limited API.) So here’s a quick intro to writing your own apps for Nokia 3310 2017. This tutorial assumes a basic level of knowledge when it comes to Java.
These steps work for Windows and have been tested on a Windows 10 VM. I strongly recommend following them on a virtual machine, as there’s going to be a certain amount of outdated software involved.
Can you use Eclipse, Netbeans or any other IDE? Possibly. However, after spending a considerable amount of time trying to set it up reliably I gave up. The IDE shipped with the J2ME SDK 3.0 was good enough for me.
Start the Java ME Platform SDK 3.0 app (henceforth: IDE). From now on, you need to create your new project,
Go to file -> New Project. Choose “MIDP Application” from the list. Set your project name to something (e.g. HelloWorld) and change the project location if you so desire. Leave “Set as Main Project” and “Create Hello MIDlet” as is. On the final step, the emulator platform should be set to CLDC…SDK 3.0. The version of the device configuration and profile are left to you – I went with the defaults. Hit finish and wait a moment.
Running is fairly simple - you have the big green Play button on the top bar.
What if you decided that you wanted to remain the autogenerated class from HelloMIDlet to, say, HelloWorld? Even if you change the class name and its definition, the project is not going to run. In such a case right-click your project, select Properties and go to Application Descriptor. There, under MIDlets, you still have your old class (by now probably marked in red). Select it to remove it from the list, then click Add and confirm the choice of the new MIDlet class. Now the app should start without any issues.
J2ME apps don’t have a main function; instead, one of the classes has to inherit from the MIDlet class and implement the CommandListener interface. The blueprint for that class has to include the following:
Let’s try to make a simple app that takes in the user’s name and puts it on screen.
We’re going to need the following fields:
private Display display;
private TextField field;
private TextField greeting;
private String name;
private Command writeName;
private Command exit;
private Form form;
Forms are the views of the app, so let’s start with the drawing out our form in the constructor. To create a new form, simply create a new Form object:
form = new Form("My first form");
To add new elements to a form, you define them and them append them to the form, for example
field = new TextField("Tell me your name", "", "30", TextField.ANY);
form.append(field);
greeting = new TextField("", "Hello, person!", 30, TextField.UNEDITABLE);
form.append(greeting);
and then append some actions to it
writeName = new Command("What's my name?", Command.OK, 0);
form.addCommand(writeName);
exit = new Command("Exit", Command.EXIT, 0);
form.addCommand(exit);
Finally, set the command listener to listen for any commands in the app:
form.setCommandListener(this);
and then set the display to show the form:
display = Display.getDisplay(this);
display.setCurrent(form);
We want the usual subjects: startApp(), pauseApp() and destroyApp(), but we’ll leave them as is. In the commandAction we’re going to read the name from the textField and then invoke the name-writing function, as well as define the exit:
public void commandAction(Command c, Displayable s) {
if (c.getCommandType() == Command.OK) {
name = field.getString();
writeName(name);
}
if (c.getCommandType() == Command.EXIT) {
destroyApp(false);
notifyDestroyed();
}
}
Finally, we need to define what writeName does behind the scenes, which is just a simple replacement:
private void writeName(String name) {
greeting.setString("Hello, " + name + "!");
}
All in all, this should give us the following code.
Here are some things I’m not going to get into details here but might be worth your while.
If you need to keep data in-between application runs, use the RecordStore.
It is possible to import third-party libraries; you can add them by right-clicking the project, selecting Properties and going to the Libraries and Resources tab.
Note, however, that if the library is not extremely small you will have to manually take in the classes you might need for it to build and run on the emulator device, which is time-consuming and definitely not fun. (I’m looking at you, the “lightweight” BouncyCastle API.)
It is possible to run J2ME on modern device; a good example is the 2017 Nokia 3310 Refurbished. Once you’ve built and tested the app on the emulator, copy the jar and jad files available at dist/ to your device (in the case of this Nokia you need a memory card), then navigate to that directory on the phone and install your app.
All in all, writing these apps for basic things is fairly easy, but gets insanely messy after a while. I can only imagine how hard it gets on really capped devices (Nokia 3310 2017 has a ~360MHz CPU; a lot of J2ME-running devices these days are micro-controllers with around 40-50MHz.)
]]>All tasks are intro-level CTF-wise, but require basic knowledge of API use, session management, HTTP communication and modern Web development in general. The fastest solve by my CTF team colleague was 5m 36s, but I’d expect it to take anywhere between half an hour and a good afternoon to someone with the grasp of the listed concepts but not much practice in CTFs or penetration tests.
Have fun and hack responsibly!
]]>Thus, the moneyprintergobrr pip package was born.
Installation:
pip install moneyprintergobrr
Usage:
from moneyprinter.moneyprinter import MoneyPrinter
MoneyPrinter().go() ## prints out 'brrr'
Actual usage:
Use it instead of printing out whatever you usually use for debugging.
Sources:
]]>I am an engineer now! I defended my thesis on Quality Assurance in Web and mobile applications a little over a month ago. Thus, I am now legally allowed to say “trust me, I’m an engineer”. Not much has changed, really, but I’m happy to have done it sooner rather than later.
I celebrated the above-mentioned achievement by taking a week off to just play video games and do nothing productive. Today’s actually my first day after that break and I’m happy to say (a) I finally finished Persona 5 after whooping 90 hours (started before the holiday, though, I didn’t spend 90 hours just playing one game) and immediately started an NG+ to have a brain cleanser ready whenever I might require it, (b) I fulfilled my long-time dream of creating my own The Sims 4 mod: now it’s possible to play Kerbal Space Program in the game and boost your rocket science skill this way:
I’ve tried really hard to use Windows 10 as my main private-laptop OS and it took me about two weeks to give up on it. Along the way, I’ve discovered just how stupid it is (note that this link is for a Windows 7 error, but the same issue applies to Windows 10, it just has a different code I can’t seem to be able to locate right now).
Speaking of stupid, I’m doing a Master’s degree! It took me a while to decide but eventually I settled for doing it and will, if everything goes well, do my thesis on attacks on computer vision systems (hopefully on both adversarial examples in deep learning and old-school, likes Viola Jones, oriented gradients, etc). I am equally terrified by and excited about the topic.
Finally, speaking of excitement, I switched jobs in October and am now well onto my way to ~world domination~ happiness and professional fulfillment. Also, can’t imagine going back to on-site work… I have all the tea I need right there in my kitchen :)
So, what’s next on the blog?
Cheers!
]]>Note: this is a 2014 argumentative essay I had to write for my classes. I couldn’t choose the subject I wanted, so I settled on the linguistical aspects of a certain word. I was very contrarian back then. :)
The word fuck is used and abused by people, in contexts way exceeding the primary meaning of “to have sexual intercourse with (someone)” (OxfordDictionaries.com). Thus, it is fairly understandable why the word itself has been the topic of research of different branches of humanities, such as linguistics, culture studies, sociology, and more, united under the common name of scatolinguistics (Zwicky, 1992:ix).
The major points of investigation seem to be fuck’s etymology and first recorded uses, the linguistic analysis of its syntactic and semantic properties, and its appearances in popular culture.
There are numerous myths and misconceptions about the origins of fuck circulating in the media. Contrary to popular urban legends, the word fuck is not an acronym for “Found Under Carnal Knowledge”, “For Unlawful Carnal Knowledge” or, better yet, “Fornication Under Consent of the King”; all these explanations emerged in the 1960s and 1970s in popular newspapers, but there is no grain of truth to them (Sheidlower, Black, 2009:i). Fuck is not, in fact, an Anglo-Saxon word at all.
Instead, etymologists point out to borrowings from either French and Latin or Old Norse, due to both the historical relation between those languages and English in the Middle Ages and the similarity of forms - French foutre, Latin futuere, Old Norse fukja, Middle Dutch fokken, Swedish focka, etc (Hughes, 2006:188). Sheidlower and Black (2009:ii-iii), however, oppose to the French/Latin roots hypothesis and are particularly of the belief that the word was borrowed from Low German, Flemish or Dutch.
Not only is the etymology of fuck unclear, but also the date of its premiere appearance in written sources arouses considerable doubt. Buck (1949) writes about one “John le Fucker”, a name that supposedly dates back to 1278, but there is neither the source of the name nor reference provided, and the name itself could, in fact, have nothing to do with fuck itself (Sheidlower, Black; 2009:ii). According to Hughes’ An Encyclopaedia of Swearing, fuck as a functional word first appeared in a 1503 poem by a Scot poet and aristocrat, William Dunbar, and it is also recorded in the 1540s as a part of a flyting, that is, a “verbal contest of insult and obscenity” (2006:189). In England, the first instances of fuck seem to have occurred at the brink of the 16th century, with Sheidlower and Black citing two cases: an Anglo-Latin poem from 1475, Flen flyys, and an annotation on a margin, “O d fuckin Abbot”, made by an anonymous monk in 1528 (2009:ii-iii). A popular teenage saying “I don’t give a fuck” may seem to date back to a 1790ish poem by George Tucker, though the piece itself had not been unveiled before the broader public until 1977 (Mohr, 2013a:iv).
Yet, common though it might have been in everyday speech and writing, fuck did not appear in monolingual dictionaries until 1965: it was present in neither Dr Samuel Johnson’s A Dictionary of the English Language nor in the Oxford English Dictionary, and it remained unlisted even in the 1961 edition of Webster Dictionary. This was mainly owing to the legislative as, since the mid-19th century, printing fuck could result in trial and incarceration. After the 1960 case Regina vs Penguin Books, there was a surge of obscenely-titled books; but when Kenneth Tynan dropped the first-ever “f-bomb” (that is, used the world fuck in a broadcast) on BBC, he provoked a nation-wide debate, with politicians roaring with rage and trying to persecute him for public obscenities. Meanwhile, in the United States, fuck appeared in abundance in slogans during the protests against the Vietnam War, such as “Fuck the Draft” and “Fuck the Pig” (Hughes, 2006:191). It also made its way into the scientific papers of that time, with linguists trying to establish the syntactic and semantic properties of fuck and its derivations.
A comprehensive literature search shows that there seem to be two kinds of fuck, namely, fuck(1) and fuck(2); for the sake of clarity, these will be referred to, respectively, as the copulative fuck and disapproving fuck (Dong; in Zwicky, 1992:3). While the former of them simply encompasses the meaning of engaging in sexual intercourse, the latter proves considerably more ambiguous. The striking difference between the two fucks can be observed by juxtaposing pairs of sentences (asterisks denote ungrammatical utterances):
(1) *Fuck you. (copulative)
(2) Fuck yourself. (copulative)
(3) Fuck you. (disapproving)
(4) *Fuck yourself. (disapproving) As the examples above indicate, the copulative _fuck_ requires using a reflexive pronoun for the interlocutor’s self-reference (cf. _Fuck yourself and I’ll give you a dollar._), while the disapproving _fuck_ bars the usage of reflexive pronouns (cf. *_Fuck yourself and leave me alone._). Moreover, as Dong (in Zwicky, 1992:4) observes, while imperative verb phrases can be conjoined, it cannot be accomplished when using the disapproving _fuck_:
(5) Wash the dishes and sweep the floor.
(6) *Wash the dishes and fuck you.
(7) *Fuck you and wash the dishes. This, along with an observation that the disapproving _fuck_ cannot take adverbial adjuncts (_Fuck communism_ *_on the sofa_/*_tomorrow_/*_gently_.), leads Dong to thinking that _fuck_ is, in fact, a "quasi-verb" (1992:7). Furthermore, Shad (in Zwicky, 1992:34) notes that _fucking_ (disapproving) does not behave like a typical adjective, either, as its insertion is allowed in the contexts in which other adjectives and degree adverbs cannot be used:
(8) That’s too fucking bad.
(9) *That’s too sad bad.
(10) That’s too fucking much.
(11) *That’s too very much. Likewise, Bopp (in Zwicky, 1992:61) also proves that the word cannot be used in a relative clause like any typical adjective:
(12) Turn off that playing radio.
(13) Turn off that radio which is playing.
(14) Turn off that fucking radio.
(15) *Turn off that radio which is fucking. Bopp also points to the fact that though _fucking well_ might seem to be an adverbial phrase, just like quietly, it does not in fact behave like one (in Zwicky, 1992:62):
(16) John quietly picked the lock.
(17) Quietly, John picked the lock.
(18) You fucking well took your time.
(19) *Fucking well, you took your time. To sum it up so far, _fuck_ and _fucking_ (disapproving) have been shown to represent quasi-verbs, quasi-adjectives and quasi-adverbs. However, _fucking_ displays another interesting property; namely, it can be used as an infix morpheme to emphasize the gravity of seemingly any non-monosyllabic word: _kanga-fuckin’-roo_, _Ala-fuckin’-bama_, _fun-fuckin’-tastic_. McCarthy (1982:577-589) notes that the placement of this infix depends on the prosodic structure of a given word: it is not allowed to break the stress pattern of a foot, there is a preference for placing the infixing before the primary stress in a word, etc.
The linguistic approach to the word fuck seems to suggest that, although its primary definition revolves around the copulative meaning, it is the disapproving connotation that has been drawing linguists’ attention. Both readings, however, have appeared in books, films and music throughout the centuries.
Even the oldest examples of the usage of fuck may seem somehow ambiguous in its reading. While the meaning of fuck in Flen flyys is clearly copulative (“They are not in heaven, since they fuck the wives of Ely” in Modern English), the meaning behind “O d fucking Abbot” is shrouded in mystery; it might mean that the friar’s engaging in sexual intercourse was frowned upon, or it might be the first attestation of the F-word in its disapproving meaning. Additionally, the d letter may stand for damned, which makes it even more interesting since it suggests that it was damned that was a real taboo at that time (Mohr, 2013b). And one of the founding fathers of the British literature, William Shakespeare, might not have used the word explicitly, but his works do contain numerous references to fuck. In The Merry Wives of Windsor, there is the focative case – a pun on the vocative case; a character in Henry V often uses the word firk, an Elizabethan euphemism for fuck; and there are numerous puns employing the French cognate foutre (Sheidlower, Black, 2009:12-13)
However, while fuck might have appeared or at least been hinted at in the written sources of the Middle Ages and the Enlightenment, its occurrence in the nineteenth- and early twentieth-century literature would still meet with the outbursts of public opinion, culminating in the above-mentioned case Regina vs. Penguin Books. The case itself was the public prosecution of Penguin Books for the publication of Lady Chatterley’s Lover by D.H. Lawrence, a book which contains such fine examples of the usage of the copulative fuck as “Fellows with swaying waists fucking little jazz girls” (in Sheidlower, Black, 2009:124) and “I’m not just my lady’s fucker, after all” (in Sheidlower, Black, 2009:170). Interestingly enough, while Lawrence’s book is often pointed to when discussing profanity, few remember about James Joyce’s Ulysses, which also contains numerous instances of the f-word in both the disapproving (“I’ll wring the neck of any bugger says a word against my fucking king”; in Sheidlower, Black, 2009:182) and copulative sense (“His wife is fucked yes and damn well fucked too”; in Sheidlower, Black, 2009:124).
Shocking though the above-presented examples might seem, modern cinema provides any careful observer with many more instances of the word. The movies directed by Quentin Tarantino have, for a long time, been regarded as “exceptionally vulgar”; but while Reservoir Dogs and Pulp Fiction, indeed, employ plenty of fucks in both readings, they only placed 20th and 21st, respectively, on the list of films using the f-word most often (Sheidlower, Black, 2009:22-23). Pulp Fiction’s use of fuck inspired Dr Sandhya Sundaresan of Leipzig University to claim that swearwords such as fuck play a semantic role similar to please (2014). More recently, Scorsese’s The Wolf of Wall Street made headlines for being the most obscene movie in history, and it indeed seems to be the most profane mainstream/Hollywood movie of all time, with the ballpark figure of 500 fucks as cited by Wickman (2014). This 2013 biographical drama not only uses the f-word in both readings but also pushes the boundaries of its morphological and pragmatic productivity: there are infix fucks (“Oompa-fuckin’-Loompas”), chants (“Fuck you” sung by traders) and neologisms (“Did the Emperor of Fucksville come down from fucking Fucksville?”, italics mine). The frequent use of fuck in the movie is reflected in numerous YouTube videos compiling the utterances into long chains of f-words.
All things considered, it seems fair to say that fuck transcends both time and space; it has been in constant use since its premiere appearance sometime in the Middle Ages, even if it has always been rejected by social mores. Its high morphological productivity has allowed countless people to express their innermost feelings and desires and stimulated the creative faculties of numerous artists. Finally, fuck has been the focal point of research for a substantial number of academics, allowing for the exploration of why exactly is the employing of the f-word so intrinsic to successful expression in everyday life.
Bibliography:
1. Hughes, Geoffrey. 2006. An Encyclopedia of Swearing. The Social History of Oaths, Profanity, Foul Language, and Ethnic Slurs in the English-Speaking World. Armonk, New York/London, England.
2. McCarthy, John J. 1982. Prosodic structure and expletive information.
3. Mohr, Melissa. 2013a. Holy Sh*t: A Brief History of Swearing. Oxford University Press, New York.
4. Mohr, Melissa. 2013b. A F*cking Short History of the F-Word. Huffington Post online; last accessed 30/05/15 at http://www.huffingtonpost.com/melissa-mohr/a-fcking-short-history-of_b_3352948.html.
5. Sheidlower, Jesse; Black, Lewis. 2009. The F-Word. Oxford University Press, New York.
6. Sundaresan, Sanhya. 2014. Why Swearing Is Just Like Saying ‘Please’ (Kind Of). Slate online; last accessed 30/05/15 at http://www.slate.com/blogs/lexicon_valley/2014/11/18/swearing_saying_please_two_ways_to_imply_an_extra_attitude_to_what_you_re.html.
7. Wickman, Forrest. 2014. Is Wolf of Wall Street Really the Sweariest Movie of All Time? A Slate Investigation. Slate online; last accessed 30/05/15 at http://www.slate.com/blogs/browbeat/2014/01/07/wolf_of_wall_street_sets_f_word_record_we_counted_every_last_f_bomb_in_the.html
8. Zwicky, Arnold (ed.) 1992. Studies Out in Left Field: Defamatory essays presented to James D. Cawley on his 33rd or 34th birthday. John Benjamins Publishing Company, Amsterdam.
mochawesome-report/.
Update: If you’re feelin’ fancy, there’s also a setup with Allure for more colourful test reports that your upper management is going to luv. Snatch them from allure-report/.
Since the workflow of this blog involves pushing to a branch other than master any time I have to do even the smallest of changes (and then putting up a PR and high-fiving myself ;)) it’s always a welcome change when I can manage with one click fewer. I’ve changed my post-commit hook for the project from
pelican content -o output -s pelicanconf.py && ghp-import output && git push origin gh-pages
to
pelican content -o output -s pelicanconf.py && ghp-import output && git push origin gh-pages && hub pull-request --force -b master -h gh-pages -m "Personal site update"
and can now enjoy just clicking the url and hitting Merge. If you can see this post it means it works!
And yes, I know that opening pull requests with the same title every time isn’t ideal…
]]>…but then it turns out you also have to have Leptonica installed first,
…but it doesn’t want to work off apt, so you decide to build it from source,
…but then it turns out that the Leptonica’s docs are not up to date but you somehow manage to make it work,
…but then it turns out that the Python:3.7 image is built atop Debian Stretch which has Tesseract 3.x in apt and you want Tesseract 4.x.
This is basically the story of this Dockerfile, sans the tears and sweat. Enjoy!
FROM python:3.7
RUN echo 'deb http://ftp.debian.org/debian stretch-backports main' > /etc/apt/sources.list.d/backports.list \
&& apt-get dist-upgrade \
&& apt-get update \
&& apt-get install -y \
make \
git \
g++ \
autoconf automake libtool \
pkg-config \
libpng-dev \
libjpeg62-turbo-dev \
libtiff5-dev \
zlib1g-dev \
&& mkdir /install \
&& cd /install \
&& git clone https://github.com/DanBloomberg/leptonica \
&& cd /install/leptonica \
&& ./autogen.sh \
&& ./configure \
&& make \
&& make install \
&& apt -t stretch-backports install -y tesseract-ocr \
&& apt -t stretch-backports install -y libtesseract-dev
WORKDIR /install
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ /install/app/
WORKDIR /install/app
CMD ["python", "app.py"]
PS so you might ask why not use Ubuntu:18.04 instead? to which I say:
module Flask cannot be found issues that took me a while to give up on.The version 0.1 of telegren is available on GitHub – all you need to do is to create your own bot by talking to the BotFather to get your own Bot Token, then run the setup script and finally the telegren script wherever you want to host the bot. It’s moderately stable now but if you come across any issues or want to add something extra just hit the Github issues for the project.
Don’t ask me to explain the name, though. It’s an inside joke.
]]>I recently digged back into the project due to my incessant interest in the concepts of readability tests and other measurable text features. While the more complex work is still underway (I need some data from humans, never easy), I thought the app itself could use some improvements, and at least one of them could potentially be done without touching the existing codebase. Then there’s the issue with how to calculate the readability score – and stay sane.
If you consider the “Books on this level” plot, there’s way too much data if you go for the B1 level:
This is due to the fact that there are way more books on the A2 - B2 levels than on any else or, putting it in the statistical lingo, the flesch.value vector data is skewed (left-skewed if you look at the flesch.value, and right-skewed if you consider the level). This becomes appearent when you group all the books by the level and count your chickens:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
book_list = pd.read_csv('book_list.csv', sep=',', header=0)
sns.set(style="ticks")
book_list.head()
| gutenberg.id | title | author | gutenberg.author.id | gutenberg.bookshelf | flesch.value | flesch.grade | level | average.goodreads.rating | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 17621 | One Day More: A Play In One Act | Conrad, Joseph | 125 | One Act Plays | 90.418363 | 5 | A1 | 2.90 |
| 1 | 28554 | Beyond Lies the Wub | Dick, Philip K. | 33399 | Science Fiction | 90.431191 | 5 | A1 | 3.81 |
| 2 | 28644 | Beyond the Door | Dick, Philip K. | 33399 | Science Fiction | 93.401235 | 5 | A1 | 3.06 |
| 3 | 32032 | Second Variety | Dick, Philip K. | 33399 | Science Fiction | 91.175281 | 5 | A1 | 3.96 |
| 4 | 41562 | The Hanging Stranger | Dick, Philip K. | 33399 | Science Fiction | 91.919921 | 5 | A1 | 3.90 |
book_list.groupby('level').size()
level
A1 14
A2 155
B1 187
B2 90
C1 22
C2 10
dtype: int64
Another thing that becomes apparent when you plot the flesch.value (click for more info on that) against the average.goodreads.rating (pulled from the Goodreads API) is that it’s also left-skewed (right-skewed on the plot below because of the nature of the Y axis).
sns.jointplot(x='flesch.value',y='average.goodreads.rating',data=book_list,kind='hex')
<seaborn.axisgrid.JointGrid at 0x7fd0b6787860>
What I get from this is that, essentially, the rating may be a useless metric (“every product and business on earth is assigned a numerical rating from 1 to 5 on some site, but everything ends up at 3.7 +/- 0.1, so it means nothing, but we just keep doing it because our first goal is quantification as such, accuracy and precision be damned” to quote Gary Bernhardt). But I promised to myself not to dwell on the past and so I’m not changing the logic of the app, and I shall just pick fewer books for each grade level to make it look nicer on the actual plot. Also, these are the cream of the crop volumes, so maybe they’re just really good on average? Or maybe people are afraid to rate them too low?
But how do I pick only some of the books in such a way that I get a sample of each readability level (and Goodreads rating)? Given the plot above, there are some really hot regions (think 3.7 stars, 78 on the Flesch-Kincaid scale) but the rest could potentially stay as it is. As this is more for the demonstration purposes and not some actual sample-sized statistical research, the simplest way would be to first bucket the books in an even more fine manner (90-88 flesch.value, 88-86 flesch.value, and so on) and then from each such bucket, if there is more than one book, choose on the basis of their Goodreads rating (how varied it is).
bins_flesch = np.linspace(0,100,num=101)
cuts_flesch = pd.cut(book_list['flesch.value'], bins_flesch)
bins_rating = np.linspace(0,5,num=11)
cuts_rating = pd.cut(book_list['average.goodreads.rating'], bins_rating)
groups = book_list.groupby(by=[cuts_flesch, cuts_rating])
groups.size().head()
flesch.value average.goodreads.rating
(26.0, 27.0] (3.0, 3.5] 1
(43.0, 44.0] (3.5, 4.0] 2
(44.0, 45.0] (3.5, 4.0] 1
(46.0, 47.0] (3.5, 4.0] 1
(47.0, 48.0] (2.5, 3.0] 1
dtype: int64
Let’s assume that for each of these buckets, the goal would be to have a single element in it, and this element would be as close to the average of that particular bucket as possible. Note that this is going to be a pretty small data frame (I’m starting with <500 records) so performance isn’t an issue here. Normally you’d want to pre-allocate indices, and stuff like that.
res_df = pd.DataFrame(columns=list(book_list.columns.values))
# https://stackoverflow.com/questions/2566412/find-nearest-value-in-numpy-array
def find_nearest(array, value):
array = np.asarray(array)
idx = (np.abs(array - value)).argmin()
return array[idx]
els = []
for key, item in groups:
g = groups.get_group(key)
g_mean = g['average.goodreads.rating'].mean()
g_mean_choice = find_nearest(g['average.goodreads.rating'], g_mean)
g_choice = g[g['average.goodreads.rating'] == g_mean_choice]
f = [res_df, g_choice]
res_df = pd.concat(f)
res_df.head()
| gutenberg.id | title | author | gutenberg.author.id | gutenberg.bookshelf | flesch.value | flesch.grade | level | average.goodreads.rating | |
|---|---|---|---|---|---|---|---|---|---|
| 469 | 15 | Moby Dick | Melville, Herman | 9 | Best Books Ever Listings/Adventure | 26.599868 | >= 16 (college graduate) | C2 | 3.47 |
| 471 | 16833 | Auguste Comte and Positivism | Mill, John Stuart | 1705 | Philosophy | 43.411004 | >= 13 (college) | C2 | 3.65 |
| 476 | 18188 | Homer and Classical Philology | Nietzsche, Friedrich Wilhelm | 779 | Philosophy | 44.382562 | >= 13 (college) | C2 | 3.84 |
| 468 | 2009 | The Origin of Species by Means of Natural Sele... | Darwin, Charles | 485 | Harvard Classics/Biology | 46.535282 | >= 13 (college) | C2 | 3.97 |
| 470 | 35421 | A System of Logic: Ratiocinative and Inductive... | Mill, John Stuart | 1705 | Philosophy | 47.673194 | >= 13 (college) | C2 | 2.82 |
cuts_flesch_2 = pd.cut(res_df['flesch.value'], bins_flesch)
cuts_rating_2 = pd.cut(res_df['average.goodreads.rating'], bins_rating)
groups = res_df.groupby(by=[cuts_flesch_2, cuts_rating_2])
groups.size().head()
flesch.value average.goodreads.rating
(26.0, 27.0] (3.0, 3.5] 1
(43.0, 44.0] (3.5, 4.0] 1
(44.0, 45.0] (3.5, 4.0] 1
(46.0, 47.0] (3.5, 4.0] 1
(47.0, 48.0] (2.5, 3.0] 1
dtype: int64
Now there should be just 1 book per bucket. The final result is 142 books (res_df.shape[0]). Plugging this into the existing app gives us the following result:
Another thing I wanted to touch upon is the readability score calculation – which is pretty hard. To calculate the Flesch-Kincaid readability score you need to correctly parse the text into sentences, words and syllables. The raw implementation of the formula would be as follows:
def calculate_flesch_score(no_of_sentences, no_of_words, no_of_syllables):
return 206.835 - 1.015 * (no_of_words / no_of_sentences) - 84.6 * (no_of_syllables / no_of_words)
Even leaving aside the issue of what is a word vs what is a sentence (take the Inuit languages with their extreme agglutination), determining the syllable count of a text is not that easy. You can use CMUDict, which is conveniently available via the nltk package, but there are a lot of words missing from there. Then there’s Pyphen but it’s pretty error-prone. I also came across the Readability package which also has its specific requirements pertaining to the pre-processing of the text.
Let’s see how all these perform on the benchmark of “Heart of Darkness”, one of the rare cases of the books I hate so much, any time I see a copy of it somewhere I want to throw it out of the closest window:
import nltk
from nltk import word_tokenize, sent_tokenize
from nltk.corpus import cmudict
import readability
import pyphen
import re
f = open('heart_of_darkness.txt')
raw = f.read()
raw.find('The Nellie')
646
raw.find('End of the Project Gutenberg EBook of Heart of Darkness, by Joseph Conrad')
211098
raw = raw[646:211098]
lower_parsed = re.sub('[^0-9a-zA-Z]+', ' ', raw).lower()
words = word_tokenize(lower_parsed)
len(words)
39064
unique_words = set(words)
len(unique_words)
5727
sentences = sent_tokenize(raw)
len(sentences)
2417
This gives me 39064 words (5727 of them unique) and 2417 sentences. Let’s see how many of these words can be found in the CMUdict:
cmudict = cmudict.dict()
no_dict = []
for word in unique_words:
if word not in cmudict.keys():
no_dict.append(word)
len(no_dict)
359
This means that 359 unique words are not accounted for in the CMUdict. Let’s see how often they appear in the whole book:
c = 0
for unique_word in no_dict:
for word in words:
if unique_word == word:
c += 1
print(c)
507
That’s a total of 507 words out of 39064, or ~1.3% of all the words in the book. Since this is a quick and dirty solution let’s say I can live with this fact, and move on. The plan is as follows:
The join-fu and error handling is due to the CMUdict sometimes having two different pronunciation of the same word (obviously, but not useful to me at that moment).
lookup_dict = {}
reg = re.compile(r'[^\d]+')
for word in unique_words:
if word not in cmudict.keys():
lookup_dict[word] = 1
else:
try:
lookup_dict[word] = len(reg.sub('', ''.join(cmudict[word][0])))
except TypeError:
lookup_dict[word] = len(reg.sub('', ''.join(cmudict[word][0][0])))
n_syl = 0
for word in words:
n_syl += lookup_dict[word]
print(n_syl)
52487
52487 sounds a bit low, but let’s see how this translates into the score:
print(calculate_flesch_score(len(sentences), len(words), n_syl))
76.76050250923326
Another implementation, now fetching the syllable count with Pyphen:
import pyphen
dic = pyphen.Pyphen(lang='en_UK') # In reality, Conrad was a Pole.
Pyphen returns a hyphenated string; I’m going to count these hyphens and add 1 to get the predicted number of syllables.
lookup_dict = {}
reg = re.compile(r'[^-]+')
for word in unique_words:
lookup_dict[word] = len(reg.sub('', word))+1
n_syl = 0
for word in words:
n_syl += lookup_dict[word]
print(n_syl)
39064
This gives me 39064 syllables, which is even more dubious.
print(calculate_flesch_score(len(sentences), len(words), n_syl))
105.8303827058337
This makes no sense whatsover. A score of 105 would be for sentences in the style of “I see you.” (essentially, a lot of single-syllable words in really short sentences), and this book is far from that. Let’s see the final implementation via the Readability package:
import readability
sentences = [s.replace('\n', ' ') for s in sentences]
text = '\n'.join(sentences)
results = readability.getmeasures(text, lang='en')
print(results['readability grades']['FleschReadingEase'])
82.8232249133162
The score of almost 83 again seems a bit too high but it’s at least close to the one calculated with the CMU dictionary. Finally, let’s see how the treetagger-based implementation performed last year:
book_list[book_list['title']=='Heart of Darkness']
| gutenberg.id | title | author | gutenberg.author.id | gutenberg.bookshelf | flesch.value | flesch.grade | level | average.goodreads.rating | |
|---|---|---|---|---|---|---|---|---|---|
| 219 | 219 | Heart of Darkness | Conrad, Joseph | 125 | Best Books Ever Listings/Movie Books | 77.935178 | 7 | B1 | 3.42 |
The treetagger version has a similar score of ~78. Maybe I just really hate this book…
]]>