Thursday, November 2, 2017

Abusing COM for tightly coupled process interaction

Twice in my career, I had to deal with unreliable third party algorithm libraries in a server situation. There's a service type program that follows a general request/response pattern. Processing a request involves calling a third party library that I don't control and that crashes far too often for comfort. The service must survive the crash, log it, and emit an error response.

Both the server and the library are native code, so a global try/catch around the library call is not really an option. So this calls for a dispatcher/worker architecture; the service receives requests and routes them to worker processes, one request at a time. If a worker crashes, the service will know and act accordingly.

One of the projects where I had to deal with this was on Linux; that's a story for another day. The other one was on Windows, and that's what I would like to discuss.

So, dispatcher/worker communication in Windows. It all hinges on the choice of an interprocess communication mechanism. I'd like an IPC that:

  • Reliably detects server crashes
  • Is message-based as opposed to stream-based
  • Has a built-in datatype marshaling logic

Component Object Model (COM) comes to mind. The worker program would be the COM server with a single object, the dispatcher would instantiate the server object and call its methods. Each request translates into one or more COM method calls. Server crash detection - check. Built-in marshaling - check. But there's a wrinkle. A COM out-of-process server is not supposed to run multiple instances. Here's how COM usually works:

  • A server executable is listed in the registry under the CLSID
  • A client calls CoCreateInstance() with that CLSID
  • The run-time starts the server executable
  • The server executable calls CoRegisterClassObject() for the CLSID
  • The run-time calls the object factory

If a subsequent call for the same CLSID comes, the run-time would reuse the same server process rather than starting a new one.

Also, the loosely coupled nature of COM is a bit of an overkill for my scenario. I never meant to expose my worker program to clients other than my dispatcher. The whole COM machinery for making servers exposed and user friendly to third party clients is irrelevant to my case.

So, how can we have a COM client creating multiple, identical COM objects running in different processes? Running Object Table (ROT) to the rescue. COM servers can publish their objects in a global repository, identified by arbitrary monikers. So the idea is:

  • The dispatcher starts multiple worker processes
  • Each gets a unique integer parameter (a cookie) via the command line
  • The worker registers an object in the ROT, identified by the cookie
  • The dispatcher retrieves that object
This is different from the regular object creation protocol. The worker program has no object factory (since it's only running exactly one object). There's no need to register the server in the registry. The object needs no CLSID. As for the interface, in my case, I'd use raw IDispatch, so there's no need for any marshaling code, either. My dispatcher/worker exchange protocol can be perfectly served by passing an array of VARIANTs both ways.

The only addition to that protocol is that the dispatcher needs to know once the worker's COM object is available in the ROT. I did that with a named event object, where the name contains the cookie. Once the worker starts up and registers its object, it would set the event. Maybe a short sleep on the dispatcher side would accomplish the same, but this is both faster and safer.

In my case, the worker can only process one request at a time, so the worker doesn't need to be multithreaded. So the CoInitialize() call in the worker would specify COINIT_APARTMENTTHREADED, and then the worker's WinMain() would have to run a message loop.

Now, dealing with the worker process crashes. There can be three kinds:
  1. During process startup
  2. During the request
  3. Between the requests
The first one is rather easy. I mentioned that the dispatcher starts the worker and waits for the "I'm ready" event. Make that a wait for two objects, the event handle and the worker process handle, and see if the process terminates before the event is set. If it does, that means a startup crash.

If the process crashes during the request, COM will report an error. The question is, which one? After some extensive testing with numerous crashes, I've identified the following HRESULT values:
Either of those means the server process terminated during the COM call, one way or another.

If the process terminates between requests, the next COM call would return HRESULT_FROM_WIN32(RPC_S_SERVER_UNAVAILABLE).

What are the quit conditions for the worker? One can implement a "please quit" method, and tell all workers to quit during dispatcher shutdown. A worker might quit when the client disconnects (e. g. the main object is released to zero). It may quit after a timeout of inactivity. In my implementation, I'd pass the PID of the dispatcher to the workers, and made them quit if the dispatcher terminates. The message loop becomes a MsgWaitForMultipleObjects() loop, with the objects being the parent process handle.

Rather than post the code here, I've published it as a Gist. The gist contains a sample worker, a sample dispatcher that creates several threads and calls the worker on each of them. The worker crashes at random with an access violation, but the dispatcher is handling that gracefully.

There's no ATL dependency in the project. There's a Native COM reference in the dispatcher, but that can be easily avoided, if dependencies are a problem. I've compiled it with Visual Studio 2017. Dispatcher is meant to be a console project (it prints some lines), while Worker is a Windows GUI one (it has a message loop).

To summarize, this is a fun little way of making COM dance. No registration, no typelibs, no proxy-stub machinery, no ref counting. Just the bits that we like - reliable interprocess communication, cross-process error handling, friendly passing of simple-typed parameters as VARIANTs.

Monday, April 3, 2017

The saga continues

Remember, some time ago Google has removed the order amount in USD from their Merchant Console? Ever since, I've used the earnings report at the end of each month to capture that data item.

Two months in, starting with the March 2017 earnings report, that's not an option, either. The order amount is there, so is the transaction fee, but the order ID isn't.

At least the support representative seemed to agree that it was a bug.

UPDATE: they've fixed it without any announcements.

Monday, February 6, 2017

Defeating the IAP emulator

A few posts ago, I've mentioned a certain Android app that emulates valid in-app purchases on rooted Android devices. I also mentioned that this app goes as far as shorting out the digital signature check code, so that apps that do due diligence and check the IAP signature against the Google public key are fooled, too.

I've been suspecting all along that the emulator does this by tapping into the Android system library, so that the built-in signature check function returns true regardless. That seems to be the case. The emulator struck again, but this time, my app had two signature checks - the system one and a homegrown one. And the latter one was the one that correctly reported a signature mismatch.

Normally, I'd be the first one to recommend against reimplementing crypto primitives. But in this case, I do feel it's justified. Here's the code. SHA1 hashing is system provided, but the RSA signature check bits are custom. The function and its parameters are deliberately called vague names, just in case the pirate crowd goes through the trouble of introducing special-case processing for my case.

Friday, January 27, 2017

Meet the new boss, same as old boss

Google is full of surprises, aren't they?

Less than three months after they've unveiled the new, redesigned Payments Center, they've discontinued it and moved the functionality to the Play developer console. And it's not like they've moved the same pages to a different host; this is a redesign, both the internals and the UI are noticeably different, with some functionality removed, and some new bugs introduced.

On the brighter side, the new UI is more scrape friendly. It's still JavaScript-driven, with explicit protection against HTTP-only scraping. On the other hand, there's a very straightforward AJAX call that returns JSON with almost all I need to capture the order activity.

There's a glaring exception though. The November 2016 version of the Payments Center would expose a crucial number - estimated revenue in USD (more generally, the payout currency). Not anymore. Unless the order was in USD to begin with, there's no way of knowing what's my take until the end of the month.

Thursday, December 22, 2016

Encoding? Compression? You tell me...

In HTTP, there are two ways to specify compression of the content:

  • Content-Encoding:gzip
  • Transfer-Encoding:gzip
Amazing discovery of the day: command-line cURL, as of v.7.21.6, recognizes both, while the cURL functions in PHP, as of PHP 5.3.3, only respect the former.

Tuesday, December 20, 2016

Strikes again

Just a few days after I wrote about the in-app purchase simulator, someone tried it again. Not that they've succeeded; the server-side order signature check caught it. But the device-side check didn't. I'm really wondering how'd they do that.

Here's the first step: I'm going to try reimplementing the signature check by hand, and see if the fake order passes that. I suspect the IAP simulator taps into Java's built-in Signature.verify() method, making it return a hard-coded true. It wouldn't know about my homegrown implementation, obviously.

The signature algorithm that Google Play uses is well known, it's SHA1 with RSA. Normally, I'd be the first one to recommend against building your own crypto primitives, but in this case, it's probably justified, at least as the first step to counteract the fraud. I can probably still use the built-in SHA1, just need to reimplement the RSA portion. The latter is a bunch of BigInteger arithmetic.

Friday, December 2, 2016

Google Play fraud

I have a freemium Android app on the Google Play store. It's free to download, but there's a paid subset of functionality. To unlock it, one may download a separate app ("The License App"), or pay with an in-app purchase (IAP).

Such a scheme is open to many avenues of abuse. I'll try to outline some of those, and discuss the ways to combat them.

Owning without buying

The other day, a customer told me: "Here's how it's normally done: you have a license app, once they install that, the premium content is unlocked". This sounds fine, until someone grabs an APK of the license app from a rooted Android device and publishes it for the whole world to see. I have some good evidence that my LicenseApp is available outside Google Play, although I never disclosed it elsewhere in any way.

Maybe in recent versions of Android grabbing a paid APK is not as straightforward as it once was. When I was last investigating this possibility, around Android 2.2, it was trivial - once the app was installed, the APK would just sit there in the filesystem. With some hackery, one could even install Google Play on the Android emulator.

Even if it's more tricky these days, all it takes is one determined person with a rooted phone.

Buying, then returning

Google Play supports a license verification service - an app may ask the Play Market app whether it was indeed purchased there. So a well written LicenseApp would not open up the paid content right away - it should ask Play first.

Enter consumer friendly refund policy. At Google Play, during the first 2 hours a customer may undo a purchase with no help from the vendor whatsoever. So, an obvious, low tech way to get a free license would be - buy the LicenseApp, let it unlock the goodies, then return with a refund. It doesn't require a rooted device.

The only way to combat something like that is checking the license periodically, e. g. once a day, but that's rather user unfriendly. What if the device has no Internet connection and the license check fails - should one lock down the functionality then? I've decided early on that I'm not going down this road; once they buy the license, the "it's bought and paid for" state stays indefinitely.

Spoofing in-app purchases

This kind of fraud is the most technically sophisticated one. There's a certain app out there that can feed valid looking in-app purchase records to apps (I'm not linking to it; do your own homework). It requires a rooted device (naturally), but the IAP record withstands all checks, including the digital signature check. I haven't investigated closely, but they probably leverage some kind of debugging API to short out the system-provided signature verification function.

I have very convincing evidence that people have been trying this against my app. There's a simple call-home functionality on successful in-app purchase; ever once in a while, I'd get a record with an order number that doesn't look like a typical Play order number, and a signature that doesn't pass a check. However, the very fact that I receive those notifications means that the IAP has passed the signature verification on the device! As an engineer, I can't help but admire. Were my app to unlock the paid goodies upon a successful IAP, it'd fall for it.

Again, if this kind of Google Play spoofing is available for the IAP logic, it's equally possible for license verification logic.

EDIT: homegrown signature check defeats that.

All of this brings me to the key insight: don't trust the device. The only way to know if a Google Play order is genuine is by checking against the Play back-end. And that's what I've been doing all along.

When I just started selling Android apps, the payment back-end behind the store (then known as Android Market) was called Google Checkout. It had a very convenient HTTP-based API - one could get a list of orders along with their status, each order record would include customer e-mail address. So once an order went through, I'd e-mail the customer a license code which they'd use to unlock. Life was good.

Then Checkout was phased out in favor of Google Wallet. And the API was gone. One could get by by HTML-scraping the administrative console, but the customer e-mail on the order details screen was gone, too. That was the biggest source of my frustration.

I had to reengineer the purchase UI to ask the user for their e-mail after a successful purchase. Most of the time they do, sometimes they don't. I'm now looking at three valid orders that I don't know what to do with, because I have no address to send the code to.

In the latest iteration of the Wallet back-end (they're no longer calling it "Wallet", either), the only way to send a message to the customer is by cancelling and refunding the order. Some irony.

Ways Google is not helping

To their credit, Google did try to provide some measure of programmatic access to my merchant data. Unfortunately, it's always too little, too late. Functional parity with the legacy Checkout is nowhere to be seen.

I can't e-mail my customers, but they're running my app, aren't they? Can't I leverage that to deliver a message to them? Yes, but how do I know the right user? All I have from the Play/Checkout/Wallet console is an order number. Here's where Google license API could help - but it doesn't. It provides no data items that I could possibly match against the Play back-end.

The only potentially useful data item is UserID (the fifth field in the signedData parameter of verifyLicense). It's a Base64 string, which decodes to a 25 byte byte array. I did some testing on a couple of live Play apps, and here are the conclusions:

  • The first 5 bytes ("the prefix") are always the same - 00D94E1D03
  • The remaining array of 20 bytes has no discernible structure - it's not a string in any reasonable charset, it doesn't look like a structure with binary integers, the bit distribution is close to random. All in all, it looks like either a hash value or a cyphertext; the former is more likely. Incidentally, MD5 hash values are exactly 20 bytes long.
  • The hash doesn't identify neither the app package nor the user - same user, different package and vice versa would yield different UserID values. The variable portion of UserID changes completely between packages and users - no bytes are the same. All the more reason to think it's a hash.
  • The UserID doesn't change if you uninstall and reinstall the app.

Play console provides sales reports, even with programmatic access via Google Storage - but those are not nearly realtime. Also, they don't list some of the very relevant information (e. g. the estimated payout amount in US dollars), and they don't list in-progress and cancelled orders.

Google Play has an API of its own, and it even has a purchase information function, but only for IAP. Paid downloads just aren't there. Also, there's no way to get the list of all transactions - only status for a specific one, and none of the financials are reported.

Ways Google is helping

All we have left is the Google payments center,  formerly known as the Wallet Merchant Center, and before that, as the Checkout console. It gives you a realtime list of orders, an order details screen, allows searching. Its most interesting feature, and a crucial one for my scenario, is the ability to search by e-mail. The customer e-mail is not exposed neither in the UI nor in the backing data, but once you have an e-mail, you can find an order (or several) by it. Without this, my business model would be dead in the water.

The e-mail search allows for partial search - lets you find orders by username without domain, even if the domain is not It even allows for username fragment search - if you have an order from, search for either "john" or "doe" or "john.doe" will find it. It doesn't search for arbitrary substrings, though - searching for "joh" won't work. Searching for users' listed first or last name won't work, either, unless they are a part of the e-mail username.

The payments center has no API of its own. But it's a website, and in that capacity, it's open to good old techniques of HTML scraping.

There are two ways to scrape a website - HTTP only and JavaScript-enabled. In the former approach, you issue HTTP requests to the right URLs using a client library of your choosing (CURL is good), then parse HTML and retrieve the data. In the latter, you run the target site in a full blown headless browser (or even a real one) and drive it by some kind of script.

For a while, scraping the Wallet console was possible with HTTP only, then, in early November 2016, they've redesigned the site to rely heavily on JavaScript and AJAX calls. At that point, I had to switch to a full headless browser scraper. PhantomJS was my tool of choice. I can't say its perfect, but it gets the job done.