Remotely Control iPhone via SSH and VNC

This is an issue I spent far too much time time on. I stumbled upon a great new game, Threes, and had become consumed with the internal workings – how the game chooses what card to produce, where to place it, etc. In order to get some real data to work with I needed to complete a multitude of games, all while keeping track of which cards were produced and where. This is the type of menial task I have no patience for and I quickly began to seek a method to do this automatically.

I had a jailbroken phone – something you will need to do have if you are interested in any of the methods I am about to describe. This method is 99.9% effective and, if something goes wrong, you can completely fix it by restoring through iTunes. Unlocking your phone is dangerous – it has the possibility of leading to a “bricked” iPhone. Jailbreaking is not unlocking. If you’re interested in jailbreaking check out evasi0n – the latest version of their tool is working on iOS 7.0.6.

With that out of the way, here’s what you’ll need and why:

  • OpenSSH: SSH is ubiquitous method of remotely accessing another system through the command line. This is how we’ll interact with the iPhone and send it the commands we want.
  • SimulateTouch: This package allows you to (you guessed it) send touch events to your iPhone using command line tools.
  • Optional Veency: This is a VNC server that runs on the iPhone. VNC is another tool that allows you to remotely access devices. However, unlike SSH, it allows you to see the screen of the remote device and send things like touch and keyboard events. But I thought we were using SimulateTouch to send touch events? Yes, that’s right. Though Veency can be used this way as well I’ve found it to be far less reliable than the SimulateTouch library. Sometimes the commands I send are just lost in the ether.
  • Optional Activator: This allows you to set up a handful (read: a ton) of custom gestures and actions that can do anything from launch applications to disable WiFi to restart your iPhone. It also, conveniently, has a command line interface which can be used to activate any of its abilities via SSH – no gestures needed.

That’s it for the software! Everything I just listed can be downloaded, for free, through Cydia. Cydia is installed automatically when you jailbreak and is the de facto way of installing 3rd party software.

So, now that we have all these tools – how do we use them?! I’m a big python buff so all of my examples are going to be in python. You can use any language you like, but you’ll have to interpret the mysterious code I provide you.

Before I start with examples, here’s the biggest pitfall of our setup:

We don’t have a way to view the current screen in applications that run at 30 FPS or more. Veency simply can’t keep up and provides a distorted display. I’ll explain how I’ve been getting around this in my examples below.

Connecting via SSH:

sshClient = spur.SshShell(hostname=IP_ADDRESS, username='root', password=SSH_PASSWORD)

I’m using Spur as my SSH interface. You can use whatever you want. We use the username ‘root’ because we want full access to the iPhone filesystem. The default SSH_PASSWORD is ‘alpine’ (and has been forever) – I recommend you change this as it is somewhat of a security issue.

Connecting via VNC:

vncClient = api.connect(IP_ADDRESS, VNC_PASSWORD)

Here I’m using the vncDoTool. Again, use what you want. You can set up the VNC_PASSWORD in the Veency settings.

Sending Touch Events:

if direction == 'right':
  sshClient.run(['stouch', 'swipe', '100', '300', '200', '300', '0.1', '1'])
if direction == 'left':
  sshClient.run(['stouch', 'swipe', '200', '300', '100', '300', '0.1', '1'])
if direction == 'up':
  sshClient.run(['stouch', 'swipe', '170', '300', '170', '200', '0.1', '1'])
if direction == 'down':
  sshClient.run(['stouch', 'swipe', '170', '200', '170', '300', '0.1', '1'])

The SimulateTouch library makes this dead simple. My example above shows how to make a swipe in any direction. You provide the X,Y coordinates of the start position and the end position. The last two commands are the duration of the swipe and the orientation of the device (1 = portrait).

Taking a ScreenShot:

sshClient.run(['activator', 'send', 'libactivator.system.take-screenshot'])
vncClient.mouseDown(3)
vncClient.mousePress(2)
vncClient.mouseUp(3)

My first example instructs Activator to take a screenshot. My second example uses VNC to hold down the power button, press the home button, and release the power button – all in quick succession. As you probably know, this is the built in way to take screenshots in iOS. These screenshots will be stored in your camera roll.

Accessing the ScreenShot:

IMAGE_DIR = 'ThreesImages'
SS_DIR = '/private/var/mobile/Media/DCIM/108APPLE'

# Copy the screenshot to disk over SSH
result = sshClient.run(['ls', SS_DIR, '-t'])
latest = result.output.split()[0]
screenshotFile = sshClient.open(SS_DIR + '/' + latest)
resultFile = open(IMAGE_DIR + '/board.png', 'w')
shutil.copyfileobj(screenshotFile, resultFile)
resultFile.close()
screenshotFile.close()

# Delete the screenshot
sshClient.run(['rm', SS_DIR + '/' + latest])

# Store the current board image
self.boardImage = Image.open(IMAGE_DIR + '/board.png')

This is a little more complicated. What we’re doing is listing the files in our screenshot directory by their creation time (yours may not be 108APPLE but it will be in the same DCIM directory). We grab the first of these files and assume (correctly) that it’s the screenshot we just took. We then open this file with SSH – this effectively stores the image in memory. Then we use python to open a file for writing – this is where we’ll store the screenshot. We use shutil to copy the screenshot from memory to the file location we opened. Finally, we close out both of our file streams.

In the second to last command I delete the screenshot file. Unfortunately, Apple has made their screenshot system ridiculously complicated. If you navigate to your Camera Roll you’ll find your image is still there. Click on it and a blank screen will be loaded. We’ve deleted the image file it’s trying to load but haven’t gotten rid of the image reference or the associated thumbnail that was created. I haven’t found a way around this.

The final command stores our image into the program memory so I can use it elsewhere without having to read the file from disk each time.

Well, this concludes this fairly verbose tutorial. If you have any questions I’ll do my best to answer them in the comments. I’ll leave you with close to 1000 lines of Python code for you to peruse. This is the code I came up with for automating the gameplay of the addictive and beautiful new game Threes.

import time, math, operator, os
import spur, shutil, random, copy
from vncdotool import api
from PIL import Image, ImageChops

IP_ADDRESS = '129.161.59.146'
VNC_PASSWORD = 'REDACTED'
SSH_PASSWORD = 'REDACTED'
IMAGE_DIR = 'ThreesImages'
SS_DIR = '/private/var/mobile/Media/DCIM/108APPLE'
DATA_FILE = 'recordedSequences.txt'

class NextBag:
  def __init__(self):
    nums = [pow(2,k) * 3 for k in range(12)]

    numOnes = 0
    numTwos = 0
    numThrees = 0
    nonThree = False
    numMax = 3

  def drawNum(self, drawnNum):
    pass

  def getNextPossibilities(self):
    return [3]

  def getNextProbabilities(self):
    return [1]

class ThreesGame:
  def __init__(self, vncClient, sshClient):
    self.vncClient = vncClient
    self.sshClient = sshClient

    self.imageDir = None

    # Only 1, 2 and 3 are known initially
    self.knownImages = []
    self.knownImages.append(Image.open(IMAGE_DIR + '/1.png'))
    self.knownImages.append(Image.open(IMAGE_DIR + '/2.png'))
    self.knownImages.append(Image.open(IMAGE_DIR + '/3.png'))

    # Only 1, 2, and 3 are possible initially
    self.possibleNumbers = []
    self.possibleNumbers.append(1)
    self.possibleNumbers.append(2)
    self.possibleNumbers.append(3)

    # Initialize the board
    self.boardImage = None
    self.boardImageIsFresh = False
    self.board = self.boardFromScreen()

    self.highest = 3

    self.drawBag = NextBag()
    self.next = []
    self.nextProbabilities = []

    self.nextCardFromScreen()

    self.dataString = ''
    self.dataCollection = False

  def startDataCollection(self):
    self.dataCollection = True
    board = self.board

    startState = []

    for row in range(4):
      for col in range(4):
        if board[row][col] != -1:
          startState.append(str(board[row][col]))

    self.dataString = '(' + ','.join(startState) + ')'

  def saveCardImage(self, row, col, filename):
    # If the board image isn't fresh, freshen it
    if not self.boardImageIsFresh:
      self.updateBoardImage()

    # Determine the current image index
    xCrop1 = col * 120 + 90
    yCrop1 = row * 160 + 339
    xCrop2 = xCrop1 + 100
    yCrop2 = yCrop1 + 75

    # Crop the full board image to the specific index
    indexImage = self.boardImage.crop((xCrop1, yCrop1, xCrop2, yCrop2))
    indexImage.save(filename)

  def updateBoardImage(self):
    # Take a screenshot
    # sshClient.run(['activator', 'send', 'libactivator.system.take-screenshot'])
    vncClient.mouseDown(3)
    vncClient.mousePress(2)
    vncClient.mouseUp(3)

    # Wait a second
    time.sleep(1)

    # Copy the screenshot to disk over SSH
    result = sshClient.run(['ls', SS_DIR, '-t'])
    latest = result.output.split()[0]
    screenshotFile = sshClient.open(SS_DIR + '/' + latest)
    resultFile = open(IMAGE_DIR + '/board.png', 'w')
    shutil.copyfileobj(screenshotFile, resultFile)
    resultFile.close()
    screenshotFile.close()

    # Delete the screenshot
    sshClient.run(['rm', SS_DIR + '/' + latest])

    # Store the current board image
    self.boardImage = Image.open(IMAGE_DIR + '/board.png')

    # Flip the fresh toggle
    self.boardImageIsFresh = True

  def cardFromScreen(self, row, col, printRMS = False):
    # Default return is 0
    cardNum = 0

    # Determine the current image index
    xCrop1 = col * 120 + 90
    yCrop1 = row * 160 + 339
    xCrop2 = xCrop1 + 100
    yCrop2 = yCrop1 + 75

    # Crop the full board image to the specific index
    indexImage = self.boardImage.crop((xCrop1, yCrop1, xCrop2, yCrop2))

    try:
      # Weird bug with #48 here - getcolors() returns None
      colors = indexImage.getcolors()
      blankColor = 0

      if colors is not None:
        blankColor = colors[0][1]

      # If image is this color we know the index is blank
      if blankColor == (187, 217, 217):
        cardNum = -1
      else:
        # Loop through each possible board image
        for index in range(len(self.knownImages)):
          if self.knownImages[index] is not None:
            # Get a histogram of the diference between the two images
            h = ImageChops.difference(self.knownImages[index], indexImage).histogram()

            # Run a RMS on the difference
            sq = (value*((idx%256)**2) for idx, value in enumerate(h))
            sum_of_squares = sum(sq)
            rms = math.sqrt(sum_of_squares/float(indexImage.size[0] * indexImage.size[1]))

            # if printRMS:
            #   ImageChops.difference(self.knownImages[index], indexImage).show()
            #   print rms

            # If the RMS is under 99 the images are 'the same'
            if rms < 99:
              cardNum = self.possibleNumbers[index]

              continue
    except TypeError, e:
      self.boardImage.show()
      indexImage.show()
      print indexImage.getcolors()
      print xCrop1, yCrop1, xCrop2, yCrop2
      print self.dataString
      raise e

    return cardNum

  def nextCardFromScreen(self):
    nextImage = self.boardImage.crop((320, 140, 321, 141))
    nextImageColors = nextImage.getcolors()[0][1]

    if nextImageColors == (102, 204, 255):
      self.next = [1]
      self.nextProbabilities = [1]
    if nextImageColors == (255, 102, 128):
      self.next = [2]
      self.nextProbabilities = [1]
    if nextImageColors == (254, 255, 255):
      self.next = self.drawBag.getNextPossibilities()
      self.nextProbabilities = self.drawBag.getNextProbabilities()

  def boardFromScreen(self):
    if not self.boardImageIsFresh:
      self.updateBoardImage()

    curBoard = [[0 for i in range(4)] for j in range(4)]

    for row in range(4):
      for col in range(4):
        curNumber = self.cardFromScreen(row, col)

        if curNumber == 0:
          print 'row:' + str(row) + 'col:' + str(col)
          self.cardFromScreen(row, col, printRMS = True)
          print self.dataString
          raise Exception('Could not determine card value')

        curBoard[row][col] = curNumber

    return curBoard

  def boardFromMove(self, direction, startBoard = None):
    if not startBoard:
      curBoard = copy.deepcopy(self.board)
    else:
      curBoard = copy.deepcopy(startBoard)

    hasChanged = False
    localChange = False

    if direction == 'left':
      for row in range(4):
        for col in range(3):
          if localChange:
            curBoard[row][col] = curBoard[row][col + 1]

          elif (curBoard[row][col] == -1) | (curBoard[row][col] == 0):
            if (curBoard[row][col + 1] != -1) & (curBoard[row][col + 1] != 0):
              curBoard[row][col] = curBoard[row][col + 1]
              localChange = True
              hasChanged = True

          elif (curBoard[row][col] + curBoard[row][col + 1]) == 3:
            curBoard[row][col] = 3
            localChange = True
            hasChanged = True

          elif (curBoard[row][col] == curBoard[row][col + 1]) & (curBoard[row][col] > 2):
            curBoard[row][col] = curBoard[row][col] * 2
            localChange = True
            hasChanged = True

        if localChange:
          curBoard[row][3] = 0

        localChange = False

    if direction == 'right':
      for row in range(4):
        for col in reversed(range(1, 4)):
          if localChange:
            curBoard[row][col] = curBoard[row][col - 1]

          elif (curBoard[row][col] == -1) | (curBoard[row][col] == 0):
            if (curBoard[row][col - 1] != -1) & (curBoard[row][col - 1] != 0):
              curBoard[row][col] = curBoard[row][col - 1]
              localChange = True
              hasChanged = True

          elif (curBoard[row][col] + curBoard[row][col - 1]) == 3:
            curBoard[row][col] = 3
            localChange = True
            hasChanged = True

          elif (curBoard[row][col] == curBoard[row][col - 1]) & (curBoard[row][col] > 2):
            curBoard[row][col] = curBoard[row][col] * 2
            localChange = True
            hasChanged = True

        if localChange:
          curBoard[row][0] = 0

        localChange = False

    if direction == 'up':
      for col in range(4):
        for row in range(3):
          if localChange:
            curBoard[row][col] = curBoard[row + 1][col]

          elif (curBoard[row][col] == -1) | (curBoard[row][col] == 0):
            if (curBoard[row + 1][col] != -1) & (curBoard[row + 1][col] != 0):
              curBoard[row][col] = curBoard[row + 1][col]
              localChange = True
              hasChanged = True

          elif (curBoard[row][col] + curBoard[row + 1][col]) == 3:
            curBoard[row][col] = 3
            localChange = True
            hasChanged = True

          elif (curBoard[row][col] == curBoard[row + 1][col]) & (curBoard[row][col] > 2):
            curBoard[row][col] = curBoard[row][col] * 2
            localChange = True
            hasChanged = True

        if localChange:
          curBoard[3][col] = 0

        localChange = False

    if direction == 'down':
      for col in range(4):
        for row in reversed(range(1, 4)):
          if localChange:
            curBoard[row][col] = curBoard[row - 1][col]

          elif (curBoard[row][col] == -1) | (curBoard[row][col] == 0):
            if (curBoard[row - 1][col] != -1) & (curBoard[row - 1][col] != 0):
              curBoard[row][col] = curBoard[row - 1][col]
              localChange = True
              hasChanged = True

          elif (curBoard[row][col] + curBoard[row - 1][col]) == 3:
            curBoard[row][col] = 3
            localChange = True
            hasChanged = True

          elif (curBoard[row][col] == curBoard[row - 1][col]) & (curBoard[row][col] > 2):
            curBoard[row][col] = curBoard[row][col] * 2
            localChange = True
            hasChanged = True

        if localChange:
          curBoard[0][col] = 0

        localChange = False

    return curBoard

  def updateNewCard(self, direction, board = None):
    if not board:
      board = self.board

    if direction == 'left':
      for row in range(4):
        if board[row][3] == 0:
          curNumber = self.cardFromScreen(row, 3)

          if curNumber == 0:
            print self.dataString
            raise Exception('Could not determine card value')

          if (curNumber != -1) & (self.dataCollection):
            self.dataString = self.dataString + ',' + str(curNumber)

          board[row][3] = curNumber

    if direction == 'right':
      for row in range(4):
        if board[row][0] == 0:
          curNumber = self.cardFromScreen(row, 0)

          if curNumber == 0:
            print self.dataString
            raise Exception('Could not determine card value')

          if (curNumber != -1) & (self.dataCollection):
            self.dataString = self.dataString + ',' + str(curNumber)

          board[row][0] = curNumber

    if direction == 'up':
      for col in range(4):
        if board[3][col] == 0:
          curNumber = self.cardFromScreen(3, col)

          if curNumber == 0:
            print self.dataString
            raise Exception('Could not determine card value')

          if (curNumber != -1) & (self.dataCollection):
            self.dataString = self.dataString + ',' + str(curNumber)

          board[3][col] = curNumber

    if direction == 'down':
      for col in range(4):
        if board[0][col] == 0:
          curNumber = self.cardFromScreen(0, col)

          if curNumber == 0:
            print self.dataString
            raise Exception('Could not determine card value')

          if (curNumber != -1) & (self.dataCollection):
            self.dataString = self.dataString + ',' + str(curNumber)

          board[0][col] = curNumber

  def sendMoveToDevice(self, direction):
    if direction == 'right':
      sshClient.run(['stouch', 'swipe', '100', '300', '200', '300', '0.1', '1'])
    if direction == 'left':
      sshClient.run(['stouch', 'swipe', '200', '300', '100', '300', '0.1', '1'])
    if direction == 'up':
      sshClient.run(['stouch', 'swipe', '170', '300', '170', '200', '0.1', '1'])
    if direction == 'down':
      sshClient.run(['stouch', 'swipe', '170', '200', '170', '300', '0.1', '1'])

    time.sleep(1)

    self.boardImageIsFresh = False

  def restartSequence(self):
    sshClient.run(['stouch', 'touch', '150', '300', '1'])
    time.sleep(0.1)
    sshClient.run(['stouch', 'touch', '150', '300', '1'])
    time.sleep(0.3)
    sshClient.run(['stouch', 'swipe', '100', '300', '200', '300', '0.1', '1'])
    time.sleep(1.0)
    sshClient.run(['stouch', 'touch', '300', '550', '1'])
    time.sleep(1.7)
    sshClient.run(['stouch', 'touch', '50', '50', '1'])
    time.sleep(1.0)

    # Reset variables
    self.boardImage = None
    self.boardImageIsFresh = False
    self.board = self.boardFromScreen()
    self.highest = 3
    self.dataString = ''
    self.knownImages = []
    self.knownImages.append(Image.open(IMAGE_DIR + '/1.png'))
    self.knownImages.append(Image.open(IMAGE_DIR + '/2.png'))
    self.knownImages.append(Image.open(IMAGE_DIR + '/3.png'))
    self.possibleNumbers = []
    self.possibleNumbers.append(1)
    self.possibleNumbers.append(2)
    self.possibleNumbers.append(3)

    # Start data collection again if flagged
    if self.dataCollection:
      self.startDataCollection()

  def checkForMax(self):
    board = self.board

    keepLooking = True

    for row in range(4):
      for col in range(4):
        if (board[row][col] > self.highest) & keepLooking:
          self.establishNewMax(row, col)
          keepLooking = False

  def establishNewMax(self, row, col):
    board = self.board

    newHighest = board[row][col]

    if self.highest > 3:
      self.knownImages.pop()
      self.knownImages.append(None)

    self.saveCardImage(row, col, IMAGE_DIR + '/Red' + str(newHighest) + '.png')
    self.possibleNumbers.append(newHighest)
    self.knownImages.append(Image.open(IMAGE_DIR + '/Red' + str(newHighest) + '.png'))
    self.highest = newHighest

    if self.dataCollection:
      self.dataString = self.dataString + ',h:' + str(self.highest)

  def checkForUnknowns(self):
    unknownNumbers = []
    unknownIndices = []
    board = self.board

    for index in range(len(self.knownImages)):
      if self.knownImages[index] is None:
        unknownIndices.append(index)
        unknownNumbers.append(self.possibleNumbers[index])

    for row in range(4):
      for col in range(4):
        if board[row][col] in unknownNumbers:
          index = unknownNumbers.index(board[row][col])
          self.saveCardImage(row, col, IMAGE_DIR + '/' + str(unknownNumbers[index]) + '.png')
          self.knownImages[unknownIndices[index]] = Image.open(IMAGE_DIR + '/' + str(unknownNumbers[index]) + '.png')
          unknownNumbers.remove(board[row][col])
          unknownIndices.remove(unknownIndices[index])

  def checkOneTwoDifference(self, board=None):
    if board is None:
      board = self.board

    numOnes = 0
    numTwos = 0

    for row in range(4):
      for col in range(4):
        if board[row][col] == 1:
          numOnes += 1
        elif board[row][col] == 2:
          numTwos += 1

    difNum = 0

    if numOnes > numTwos:
      difNum = numOnes - numTwos
    elif numOnes < numTwos:
      difNum = numTwos - numOnes

    if difNum > 4:
      # Take a screenshot
      # sshClient.run(['activator', 'send', 'libactivator.system.take-screenshot'])
      vncClient.mouseDown(3)
      vncClient.mousePress(2)
      vncClient.mouseUp(3)

      # Wait a second
      time.sleep(1)

      # Copy the screenshot to disk over SSH
      result = sshClient.run(['ls', SS_DIR, '-t'])
      latest = result.output.split()[0]
      screenshotFile = sshClient.open(SS_DIR + '/' + latest)
      resultFile = open(IMAGE_DIR + '/moreThanProof.png', 'w')
      shutil.copyfileobj(screenshotFile, resultFile)
      resultFile.close()
      screenshotFile.close()

      # Delete the screenshot
      sshClient.run(['rm', SS_DIR + '/' + latest])

      self.dataString = self.dataString + ',!!!(numOnes:' + str(numOnes) + 'numTwos:' + str(numTwos) + ')!!!'

  def makeMove(self, direction):
    print 'Move:', direction

    print 'Simulating move internally...',
    newBoard = self.boardFromMove(direction)
    print 'DONE'

    if newBoard == self.board:
      print 'Invalid move - returning'
      return
    else:
      oldBoard = copy.deepcopy(self.board)
      self.board = newBoard

    print 'Sending move to device...',
    self.sendMoveToDevice(direction)
    print 'DONE'

    print 'Updating board image...',
    self.updateBoardImage()
    print 'DONE'  

    print 'Checking for a new maximum...',
    self.checkForMax()
    print 'DONE'

    print 'Checking for any unidentified card images...',
    self.checkForUnknowns()
    print 'DONE'

    print 'Checking if boardFromScreen is expected',
    goodToGo = self.assertExpectedBoard()

    if not goodToGo:
      screenBoard = self.boardFromScreen()

      # The last move didn't go through... try it again
      if screenBoard == oldBoard:
        print 'OH NO! The last move didnt go through!'

        print 'Sending move to device...',
        self.sendMoveToDevice(direction)
        print 'DONE'

        print 'Updating board image...',
        self.updateBoardImage()
        print 'DONE'  

        print 'Checking for a new maximum...',
        self.checkForMax()
        print 'DONE'

        print 'Checking for any unidentified card images...',
        self.checkForUnknowns()
        print 'DONE'

        print 'Checking if boardFromScreen is expected',
        goodToGo = self.assertExpectedBoard()

        if not goodToGo:
          print '\nScreenBoard'
          self.displayBoard(self.boardFromScreen())
          print 'ReferenceBoard'
          self.displayBoard(self.board)
          print self.dataString
          raise Exception('Assert error - unexpected board (we tried)')

      else:
        print '\nScreenBoard'
        self.displayBoard(self.boardFromScreen())
        print 'ReferenceBoard'
        self.displayBoard(self.board)
        print self.dataString
        raise Exception('Assert error - unexpected board')

    print 'DONE'

    print 'Updating internal board with new card...',
    self.updateNewCard(direction)
    print 'DONE'

    print 'Determining the next card...',
    self.nextCardFromScreen()
    print 'DONE'

    print 'Checking if difference between 1s and 2s is >4...',
    self.checkOneTwoDifference()
    print 'DONE'

  def isGameOver(self, board=None):
    if board is None:
      board = self.board

    foundEmpty = False

    for row in range(4):
      for col in range(4):
        if board[row][col] == -1:
          foundEmpty = True

    isOver = False

    if not foundEmpty:
      leftBoard = self.boardFromMove('left', board)
      rightBoard = self.boardFromMove('right', board)
      upBoard = self.boardFromMove('up', board)
      downBoard = self.boardFromMove('down', board)

      if (leftBoard == rightBoard) & (leftBoard == upBoard) & (leftBoard == downBoard):
        self.displayBoard(board)
        self.displayBoard(leftBoard)
        self.displayBoard(rightBoard)
        self.displayBoard(upBoard)
        self.displayBoard(downBoard)
        isOver = True

    return isOver

  def assertExpectedBoard(self, refBoard=None):
    if refBoard is None:
      refBoard = self.board

    screenBoard = self.boardFromScreen()

    for row in range(4):
      for col in range(4):
        if refBoard[row][col] != 0:
          if refBoard[row][col] != screenBoard[row][col]:
            return False

    return True

  def scoreBoard(self, board=None):
    if board is None:
      board = self.board

    nums = [pow(2,k) * 3 for k in range(12)]
    scores = [pow(3,k+1) for k in range(12)]
    total = 0

    for row in range(4):
      for col in range(4):
        if board[row][col] in nums:
          index = nums.index(board[row][col])
          total += scores[index]

    return total

  def displayBoard(self, board=None):
    isBigger = True
    biggestNum = -1
    spacing = 3

    if board is None:
      board = self.board

    # Get the largest number on the board
    for row in board:
      for num in row:
        if num > biggestNum:
          biggestNum = num

    # Determine spacing
    while len(str(biggestNum)) > spacing:
      spacing += 1

    # Print the board
    headFoot = '-' * (5 + spacing * 4)
    mid = ('|' + '-' * (spacing)) * 4 + '|'

    print headFoot

    for i in range(len(board)):
      print '|',
      for num in board[i]:
        if num == -1:
          strNum = 'X'
        else:
          strNum = str(num)

        digits = len(strNum)
        preDigits = ' ' * ((spacing - digits) / 2)
        postDigits = ' ' * (spacing - digits - len(preDigits))

        print '\b' + preDigits + strNum + postDigits + '|',

      if i < len(self.board) - 1:
        print '\n' + mid

    print '\n' + headFoot# + '\nNext: ' + str(self.next)

  def fillBoardWith(self, numToFill, board=None):
    if board is None:
      board = copy.deepcopy(self.board)

    filledBoards = []
    zeroLocations = []

    for row in range(4):
      for col in range(4):
        if board[row][col] == 0:
          zeroLocations.append(row)
          zeroLocations.append(col)

    for index in range(len(zeroLocations)):
      filledBoards.append(copy.deepcopy(board))

      for anotherIndex in range(len(zeroLocations)):
        row = zeroLocations[0]
        col = zeroLocations[1]

        if anotherIndex == index:
          filledBoards[-1][row][col] = numToFill
        else:
          filledBoards[-1][row][col] = -1

    return filledBoards

  def touchArcadeHeuristic(self, board=None):
    if board is None:
      board = self.board

    totalScore = 0

    # +4 for each empty square
    for row in range(4):
      for col in range(4):
        if self.board[row][col] == -1:
          totalScore += 4

    # +4 for each possible combo left<->right
    for row in range(4):
      for col in range(3):
        if self.board[row][col] + self.board[row][col + 1] == 3:
          totalScore += 4
        if self.board[row][col] == self.board[row][col + 1]:
          totalScore += 4

    # +4 for each possible combo up<->down
    for row in range(3):
      for col in range(4):
        if self.board[row][col] + self.board[row + 1][col] == 3:
          totalScore += 4
        if self.board[row][col] == self.board[row + 1][col]:
          totalScore += 4

    # -1 for each low card between two high cards
    for row in range(4):
      for col in range(1,3):
        left = self.board[row][col - 1]
        right = self.board[row][col + 1]
        middle = self.board[row][col]

        if (left > middle) & (right > middle):
          totalScore -= 1

    # -1 for each low card between two high cards
    for row in range(1,3):
      for col in range(4):
        top = self.board[row - 1][col]
        bottom = self.board[row + 1][col]
        middle = self.board[row][col]

        if (top > middle) & (bottom > middle):
          totalScore -= 1

    return totalScore

  def determineNextMove(self, method):
    moves = ['left', 'right', 'up', 'down']

    # Simplest method - whichever move results in the highest score
    # Ignore the next card in the sequence
    if method == 0:
      score = []

      for move in moves:
        nextBoard = self.boardFromMove(move)
        score.append(self.scoreBoard(nextBoard))

      maxScore = max(score)

      possibleIndices = [i for i, j in enumerate(score) if j == maxScore]

      chosenIndex = possibleIndices[random.randint(0, len(possibleIndices) - 1)]

      return moves[chosenIndex]
    # Look for best score in two moves assuming the white card is always a three
    if method == 1:
      score = []

      for move in moves:
        curScore = []

        nextBoard = self.boardFromMove(move)
        filledNextBoards = self.fillBoardWith(self.next[0], nextBoard)

        for board in filledNextBoards:
          for nextMove in moves:
            nextNextBoard = self.boardFromMove(nextMove, board)
            for nnextMove in moves:
              nnextBoard = self.boardFromMove(nnextMove, board)
              for nnnextMove in moves:
                nnnextBoard = self.boardFromMove(nnnextMove, board)
                for nnnnextMove in moves:
                  nnnnextBoard = self.boardFromMove(nnnnextMove, board)
                  curScore.append(self.scoreBoard(nnnextBoard))
                  # curScore += self.touchArcadeHeuristic(nnnextBoard)
                  # curCount += 1

        # for board in filledNextBoards:
        #   for nextMove in moves:
        #       nextNextBoard = self.boardFromMove(nextMove, board)
        #       curScore.append(self.scoreBoard(nextNextBoard))

        if len(curScore) == 0:
          score.append(0)
        else:
          score.append(max(curScore))

      maxScore = max(score)

      possibleIndices = [i for i, j in enumerate(score) if j == maxScore]

      chosenIndex = possibleIndices[random.randint(0, len(possibleIndices) - 1)]

      return moves[chosenIndex]
    # Use the touch arcade method looking two deep, using only the first card and assuming three
    if method == 2:
      score = []

      for move in moves:
        curScore = 0
        curCount = 0

        nextBoard = self.boardFromMove(move)
        filledNextBoards = self.fillBoardWith(self.next[0], nextBoard)

        for board in filledNextBoards:
          for nextMove in moves:
            nextNextBoard = self.boardFromMove(nextMove, board)
            for nnextMove in moves:
              nnextBoard = self.boardFromMove(nnextMove, board)
              for nnnextMove in moves:
                nnnextBoard = self.boardFromMove(nnnextMove, board)
                for nnnnextMove in moves:
                  nnnnextBoard = self.boardFromMove(nnnnextMove, board)
                  curScore += self.touchArcadeHeuristic(nnnextBoard)
                  curCount += 1

        if curCount == 0:
          score.append(0)
        else:
          score.append(float(curScore)/curCount)

      maxScore = max(score)

      possibleIndices = [i for i, j in enumerate(score) if j == maxScore]

      chosenIndex = possibleIndices[random.randint(0, len(possibleIndices) - 1)]

      return moves[chosenIndex]

if __name__ == '__main__':
  # Connect to the iOS Device via VNC
  vncClient = api.connect(IP_ADDRESS, VNC_PASSWORD)

  # Connect to the iOS Device via SSH
  sshClient = spur.SshShell(hostname=IP_ADDRESS, username='root', password=SSH_PASSWORD)

  # Create a Threes Game
  myGame = ThreesGame(vncClient, sshClient)
  myGame.startDataCollection()

  moves = ['left', 'right', 'up', 'down']
  random.seed()

  gameOver = False

  while(not gameOver):
    nextMove = myGame.determineNextMove(1)
    myGame.makeMove(nextMove)
    gameOver = myGame.isGameOver()

    if gameOver:
      print 'over'
      print myGame.dataString

      with open(DATA_FILE, "a") as myfile:
        myfile.write('\n' + myGame.dataString)

      myGame.restartSequence()
      gameOver = False

 

Programmatically Add Default UITextField

I haven’t posted anything in a while, but I’ve been programming much more often and figured I’d share some answers to questions that had me stuck for a while. Now, if you’ve ever tried to add a UITextField purely through code you’ve probably noticed that what you get is a blank area that you can double-tap and add text to. When you add it using Interface Builder you get this pretty rectangle with rounded edges – what gives?

Well, I have no idea what gives. But what I do know is how to use a whole bunch of extra code to fix it. Here’s some nicely commented code that shows what I’m talking about:

- (UITextField *)getCustomTextField: (NSString *)placeholderText xLoc:(int)X
                               yLoc:(int)Y width:(int)width height:(int)height
{
    // Start by allocating an ugly default text field
    UITextField *textField = [[UITextField alloc] initWithFrame:CGRectMake(X, Y, width, height)];
    
    // Set font to default text field font
    textField.font = [UIFont fontWithName:@"Helvetica Neue" size:14];
    
    // If the man wants some placeholder text, give it to him
    if (placeholderText != nil)
        textField.placeholder = placeholderText;
    
    // Add the clear button while editing
    textField.clearButtonMode = UITextFieldViewModeWhileEditing;
    
    // Design our border so it looks like a "real" UITextField
    [textField.layer setBorderColor:[[[UIColor grayColor] colorWithAlphaComponent:0.4] CGColor]];
    [textField.layer setBorderWidth:0.5];
    
    // Make that sexy corner all round
    textField.layer.cornerRadius = 7;
    textField.clipsToBounds = YES;
    
    // Add padding to the box so it looks real
    UIView *paddingView = [[UIView alloc] initWithFrame:CGRectMake(0, 0, 9, 0)];
    textField.leftView = paddingView;
    textField.leftViewMode = UITextFieldViewModeAlways;
    return textField;
}

That method will return a nicely formatted UITextField. It may not look precisely like the one you get from IB, but I challenge you to get it closer. If you don’t want any placeholder text just send nil or edit the method to take no instance variables.

Answers for Pilots

Visit www.AnswersForPilots.com!

Created by Sam and James Sullivan and the winner of some awesome GoDaddy scholarship! This page will look better once James and/or Sam give me a large block of text to put here!

Windows 8 Lightscribe Disc Image

I couldn’t find any decent Lightscribe templates for a Windows 8 disk so I whipped one up myself using a mashup of others’ work. Enjoy!

[nggallery id=10]

The creator of the Windows 7 image this is based off has a Deviant Art account.

I was unable to find the creator of the Windows 8 logo I used in the image. Creator, if you stumble upon my website feel free to shoot me an email for recognition.

[download id=”7″ format=”1″]

The Inception Button on Cydia

I received an email earlier today asking if I could get the Inception Button onto Cydia seeing as Apple has the preposterous idea that such an application provides “limited functionality” (rather than limitless functionality). I figured, why not, I’d give it a go.

After some time on IRC channels with the creator of Theos and several other helpful and knowledgeable folks I’ve managed to bring the barebones version of the Inception Button to Cydia! I’d like to thank Allen of Planet-iPhones for hosting the application. Anyone who is jailbroken should be able to install the Inception Button via the default community sources.





[nggallery id=9]

Automatically Delete iTunes Duplicate Songs with iDupeNuke

UPDATE: I’m finding more and more bugs as I use this. I wouldn’t recommend using it in its current state.

The various methods by which I accumulate my music often leave me with duplicate songs in my iTunes library. Having been fed up with the task of manually removing these files I decided to create a utility to do it for me. The project took me quite a bit further into the night than I anticipated however I was able to come out with an end product I’m happy to call a 0.1 Beta.

The utility is written in Java so it should, in theory, be compatible on both Windows and OS X. That being said, the only testing I’ve done has been on a Windows 7 machine and it was quite minimal. With the latest version of Java installed you can instruct iDupeNuke to remove duplicate songs in iTunes with the following code

java -jar iDupeNuke.jar

The program should proceed to analyze and neutralize any duplicate songs in your iTunes library.

I welcome suggestions, bug reports, and generic comments as I plan to expand this project and implement some sort of GUI. I hate Swing – if anyone has a recommended alternative I’m all ears.

[download id=”6″ format=”1″]

WakeMate Reviews

Visit WakeMate Reviews

With the release of the WakeMate I decided to pull together a site to provide prospective customers with current information and reviews on the intriguing device. In addition to providing information, the site encourages people who already own a WakeMate to submit their own thoughts, comments, suggestions, and reviews.

Please feel free to post your thoughts or suggestions on the new site either in the comments here or via the WakeMate Review site itself.

The State of the Tablet Market: Q1 2011

Since the launch of the iPad in 2010 the tablet market has been developing in an interesting way. Apple had been ingenious in the years leading up to the iPad with iPhone sales consistently breaking records and defining the industry standard for the last few years. As hardware and software makers began to keep stride with the iPhones and release plethoras of Android phones along with a smattering of new BlackBerries, WebOS phones, and Windows 7 Phones Apple was able to rush the iPad to the market with a tried and tested software interface.

The iPad was a logical step that hardly anyone saw coming and even fewer people imagined could be successful. Steve Jobs recently revealed that the iPad was the idea in the first place and he rushed to make a phone out of it. The timing of the entire thing was gorgeous – not only was it able to beat the JooJoo to market (which many people thought would be a real contender) it became the clear choice for a tablet. To this day the only other real contenders are Linux or Android tablets or Windows 7 tablets. The Windows 7 tablets are a mess because, as touch friendly as that OS is, it’s not good enough to be thrown on a tablet willy nilly. The Android tablets suffer from the reverse issue – an inappropriate upscaling of a mobile OS.

Some may argue the iPad suffers from this same problem but the truth is the iPad is an entirely different experience than the iPhone. Things will start to get interesting as we see how Chrome OS might adapt on a tablet and what HP can do with WebOS (I predict a change reminiscent of iOS -> iPad OS) to make it work on a tablet.

The iPad 2 will be coming out soon likely with two cameras and the highest resolution screen on a tablet device which will push the market further in Apple’s direction and force some much needed innovation. We’re not far at all from components powerful enough that a MacBook Air-esque device will be able to boot into a more iPad-esque OS (if Apple pulls this off they’ll probably manage it without a noticeable reboot time… very doable with solid state storage) to be used as a more mobile tablet while retaining the full power of OS X either when in a docking station or on your lap as a laptop rather than in your palm as a tablet.

As it stands things are going to get very interesting in the next year and the trend shows no sign of slowing. If it doesn’t go without saying I would definitely recommend waiting until the iPad 2 at the very earliest for those of you in the market for a tablet. If you don’t like the direction Apple is heading the next best thing (disregarding any CES hooplah) is the Color Nook which runs a crippled version of Android but, in my opinion, has the right idea on the hardware front (at least until color e-ink becomes a reality).

I’ll also mention that the software developers of tablet and phone operating systems alike have to do some serious work on the notifications front before Apple plugs their obvious hole. Right now Android does it best with the pull down bar (and I think WebOS did pretty good) but there is a fine balance between user control, automation, intrusiveness, and unobtrusiveness that Apple has been searching for behind the scenes for the last few years (and slowly working towards – multitasking implementation, anyone?) and they’re bound to stumble upon it soon.

Apologies for the rough article, I’m not in a proofreading mood.

100 Rogues on Android?

UPDATE: See the current status of this project on 100 Rogues’ Blog

Wouldn’t it be just great if some aspiring developer who just completed a semester of Java downloaded the Android SDK and set to work on bringing the iOS hit – 100 Rogues – to Android?

[singlepic id=17 w=320 h=240 mode=web20 float=center]

Apache Derby and Java in Mac OS X

In order to work with Apache Derby in Mac OS X you will first need the .jar files. They can be downloaded from Apache’s site here. The current release comes in several flavors:

  • bin distribution – contains the documentation, javadoc, and jar files for Derby.
  • lib distribution – contains only the jar files for Derby.
  • lib-debug distribution – contains jar files for Derby with source line numbers.
  • src distribution – contains the Derby source tree at the point which the binaries were built.

Download whichever fits your needs – the only essential files are the jar files. Execute the following line in terminal to set a DERBY_HOME variable.

export DERBY_HOME=”Insert the location of your downloaded directory”

You can then enter the next line of code into terminal to set your classpath to include DERBY_HOME (and the two needed jar files) and the root directory.

export CLASSPATH=”$DERBY_HOME/lib/derbytools.jar:.:$DERBY_HOME/lib/derby.jar:$CLASSPATH”