Refactoring a dynamically typed language: do it safely or automatically, but not both

I was recently having a discussion about refactoring dynamically typed languages and I was struck by the amount of misconceptions that a lot of developers still have on this topic.

I expressed my point in this article from fifteen years ago(!), not much has changed, but the idea that it it impossible to safely and automatically refactor a language that doesn’t have type annotations is still something that is not widely accepted, so I thought I would revisit my point and modernize the code a bit.

First of all, my claim:

In languages that do not have type annotations (e.g. Python, Ruby, Javascript, Smalltalk), it is impossible to perform automatic refactorings that are safe, i.e., that are guaranteed to not break the code. Such refactorings require the supervision of the developer to make sure that the new code still runs.

First of all, I decided to adapt the snippet of code I used in my previous article and write it in Python. Here is a small example I came up with:

class A:
    def f(self):

class B:
    def f(self):

if __name__ == '__main__':
    if random() > 0.5:
        x = A()
        x = B()

Pretty straightforward: this code will call the function f() on either an instance of class A or B.

What happens when you ask your IDE to rename f() to f2()? Well, this is undecidable. You might think it’s obvious that you need to rename both A.f and B.f, but that’s just because this snippet is trivial. In a code base containing hundreds of thousands of lines, it’s plain impossible for any IDE to decide what functions to rename with the guarantee of not breaking the code.

This time, I decided to go one step further and to actually prove this point, since so many people are still refusing to accept it. So I launched PyCharm, typed this code, put the cursor on the line x.f() and asked the IDE to rename f() to f2(). And here is what happened:

PyCharm renamed the first f() but not the second one! I’m not quite sure what the logic is here, but well, the point is that this code is now broken, and you will only find out at runtime.

This observation has dire consequences on the adequacy of dynamically typed languages for large code bases. Because you can no longer safely refactor such code bases, developers will be a lot more hesitant about performing these refactorings because they can never be sure how exhaustive the tests are, and in doubt, they will decide not to refactor and let the code rot.

Update: Discussion on reddit.

Malware on my Android phone!

I have a confession to make that I’m not very proud of: recently, I unwittingly installed malware on my Android phone. As one of the early members of the Android team and someone who’s been using Android for about thirteen years, this was a pretty humbling and irritating event. This is what happened.

I remember how it started: I unlocked my phone and two accidental clicks led me to agree to a dialog that my brain immediately registered as suspicious. But I had other things on my mind at the time so I paid it no mind and moved on.

The next day, I picked up my phone and when I launched Chrome, I immediately noticed it was displaying a spammy URL. What’s worse: there were over ten tabs displaying similar URL’s which I was certainly not visiting before going to bed. This is when I realized what had happened.

DefCon 5

Lacking the time to do an investigation at the time, I went for an easy and temporary solution: changed my default browser from Chrome to another one and moved on. A few hours later, I found a similar set of URL’s displayed by my new default browser, which now definitely pointed the finger toward a rogue app.

DefCon 4

I went through all my apps and uninstalled and disabled a bunch of them. I also downloaded MalwareBytes and ran it, to no effect. Google Play Protect also did not notice anything suspicious on my phone. I must have missed the offending app because the URL’s kept showing.

DefCon 3

Time to go medieval. I hooked up my phone and launched adb. I started by inspecting the list of recurring tasks, but the output was so voluminous that finding anything useful was a dim prospect. I also inspected all the services and their associated applications, but then again, it was pretty much impossible to find anything suspicious even with some well targeted `greps`.

DefCon 2

I resigned myself to the brute force approach. I launched Android Studio, filtered the logcat output on “http.?://“, moved the window on the side and resumed my activities while keeping an eye on which URL’s my phone visits while I’m not using it.

It only took about an hour until one of the fishy URL’s showed up:

2021-01-09 11:01:14.651 3655-4415/? I/ActivityTaskManager: START u0 
{act=android.intent.action.VIEW dat=
 flg=0x10000000 cmp=org.adblockplus.browser.beta/} from uid 10237

Ha HA! I got you now. I have a uid, which I grepped through the output, and I finally identified my target:

2021-01-09 11:01:13.810 3655-3655/? I/EdgeLightingManager: showForNotification :
isInteractive=false, isHeadUp=true, color=0,
sbn = StatusBarNotification(pkg=com.qrcodescanner.barcodescanner user=UserHandle{0} id=1836 tag=null 
key=0|com.qrcodescanner.barcodescanner|1836|null|10237: Notification(channel=sfsdfsdfsd pri=2 contentView=null
vibrate=null sound=null defaults=0x0 flags=0x90 color=0x00000000 vis=PRIVATE semFlags=0x0 semPriority=0 semMissedCount=0))

So the package name is “com.qrcodescanner.barcodescanner“. It looks like the rogue app is disguising itself as a QR barcode scanner. I took another look at the list of my apps and sure enough, I quickly located the offending application. I uninstalled it, and a few hours later, I was happy to observe that the URL’s stopped popping up.

Back to DefCon 5

I went back to the Play Store and tried to find the application that I uninstalled but couldn’t find it, which indicates that Google probably removed it from its store some time ago. The malware I activated most likely side loaded it after I unwittingly approved its installation.

Some forensic research revealed an article from 2018 discussing such a malware QR Code Reader application, but the package and the mode of operation don’t quite match what I found. I am probably dealing with a copycat.

Looking back, I feel pretty disappointed that I had to go through all these steps to get rid of a simple scam application. What would a regular user do?

Conclusions and suggestions

  • Listing the apps installed on my phone should give me the option to sort them by “Latest installed”. I am pretty sure that if I had had this option and I had seen a QR Code Scanner installed just a few days ago, it would have immediately grabbed my attention. As it is, the way Android lists the installed apps is pretty useless for this purpose.
  • MalwareBytes was completely useless and I immediately uninstalled it when I realized this fact. The problem is that it was probably just looking for malware code signatures inside the packages instead of just looking at which apps I had installed.
  • Google Play Protect was also completely unhelpful, which was a big disappointment. First because Google certainly knows which applications they removed from their store for malware reasons, but even so, I would expect Google Play Protect to at least flag any app it finds on my phone that is not on their store. Such an app is not necessarily malware, but it should certainly be flagged.
  • Google Play Protect could also do some behavior profiling to analyze what apps are doing in the background. A service launching recurring VIEW intents on web sites in the background should have raised a flag to the system.

Zoom background #23

Zoom background #22

A flour windmill on the island of Leros, Greece.

Zoom background #21

This is what the sky looks like in the Bay Area this week. The orange glow is due to high altitude smoke from the fires that are ravaging the area. The air quality is actually not too bad because the smoke is much higher than usual.

Zoom background #20

Screen shot Flight Simulator 2020. The game receives real time weather data and this is a model of hurricane Laura as it was unfolding.

Zoom background #19

A Chip-8 emulator written in Kotlin

I’ve always wanted to write an emulator. First because I have a lot of interest in retrocomputing, and more particularly, in cracking old school Apple ][ games, but mostly because an emulator has always seemed to me to be a great mix of technical challenge with a very rewarding feeling as you make progress. So I made it my week end project to write a Chip-8 emulator.

Chip-8 is a very popular CPU to emulate and usually the first project people who are new to this exercise undertake, so it was an easy choice. The spec is short and except for a few tiny details, very clear on how to implement this CPU. It’s graphical too, which is important from a reward standpoint. I was absolutely thrilled when I saw the beginning of the welcome screen of Space Invaders appear on my screen after I had implemented just a few opcodes.

You can find the emulator on Github with all the technical details.

I am very tempted to work on a harder emulator now, from a real console. I am fluent in 6502 so maybe SNES, or even an Apple ][…

Zoom background #18

Screen shot from Assassin’s Creed: Odyssey

Interesting reads – 08/09/2020